BeagleBone Black Console Cable

I recently got a BeagleBone Black from Adafruit. Immediate access was easy using the USB interface:

sudo screen /dev/ttyACM0 115200

To disconnect, press [Ctrl][A], followed by [k], and respond [y].

I wanted to watch boot sequence, but the USB does not come online until the end of the init process. I had purchased a USB to serial adapter, but did not find an obvious path between the "Prolific Technology PL2303" converter, and the serial header on the BeagleBone. The lack of documentation should not have been a surprise: most people have no interest in this feature.

To make matters more interesting, the picture on the Adafruit website was off by one pin. Luckily, Google Images was able to find the blog of Christophe Blaess, who had posted a picture of the correct pin-out:

The blog is written in French, which I don't speak, but a picture is worth a thousand words.

PVM, aka Beowulf Cluster

I stumbled upon a little project this weekend and found myself without a Beowulf cluster to help out.  It had been several years since I'd built a computational cluster, so I noticed a few "new" gothchas.  But... before we get to the fun stuff, let's review:

• No, Beowulf is not "dead technology"
• No, Hadoop is not the perfect tool for every job

That which was once called a Beowulf cluster is actually the use of a Message Passing Interface (MPI) that when deployed across nodes, create a Parallel Virtual Machine (PVM).  Ready for this-- the current generation of Hadoop (et al) clusters, are actually MPIs, and thus PVMs.  Given network attached storage, a map-reduce cluster is, in theory, Beowulf compliant.

To set up the absolute simplest PVM, we need two nodes, with an NFS share, and a user account.  The user needs an SSH key pair distributed to all nodes such that the user can login to any machine, from any machine.  Each node's hostname must be able to resolve via DNS or /etc/hosts.  Each node's hostname and address must be statically configured, and cannot be "localhost".

The first step is to install the base package from an EPEL repo.  (I'm using Scientific Linux 6.)  The package is delivered as source and must be compiled with a minimal set of options:

yum install -y pvm --nogpgcheck
rpm -ql pvm | grep -m1 pvm3
/usr/share/pvm3

This shows us where the RPM installed the source.  The issue with this incarnation is that it is still configured for RSH rather than SSH:

export PVM_ROOT=/usr/share/pvm3/
cd $PVM_ROOT
find . -type f -exec sed -i "s~bin/rsh~bin/ssh~g" {} \;
make; make install

Unfortunately, there are still hard-coded references to RSH in some of the binary libraries, so we spoof the references with a symlink:

ln -s /usr/bin/ssh /usr/bin/rsh

Repeat these steps on all (both) nodes.

On only one of the nodes (it doesn't matter which one) validate that PVM is not running, configure the PVM_ROOT variable, and start the first instance as the non-root user:

ps -ef | awk '!/awk/ && /pvm/'
echo "export PVM_ROOT=/usr/share/pvm3" >> ~/.bashrc
echo id | pvm
pvm> id
t40001
Console: exit handler called
pvmd still running.
ps -ef | awk '!/awk/ && /pvm/'
compute <snip> /usr/share/pvm3/lib/LINUXX86_64/pvmd3

Notice that the PVM deamon launched and remained resident.  Individual commands can be piped to PVM, or an interactive console can be used.  From the same node, remotely configure the next node:

ssh pvm2 'echo "export PVM_ROOT=/usr/share/pvm3" \
                >> ~/.bashrc'
# should not prompt for a password
ssh pvm2 'echo $PVM_ROOT'
/usr/share/pvm3
ssh pvm2 'rm -f /tmp/pvm*'

The last line is the very, very, important.  From the first node, remotely start the second node:

pvm
pvmd already running.
pvm> conf
conf
1 host, 1 data format
   HOST     DTID     ARCH   SPEED       DSIG
   pvm1    40000 LINUXX86_64    1000 0x00408c41
pvm> add pvm2
add pvm2
1 successful
   HOST     DTID
   pvm2    80000
pvm> conf
conf
2 hosts, 1 data format
   HOST     DTID     ARCH   SPEED       DSIG
   pvm1    40000 LINUXX86_64    1000 0x00408c41
   pvm2    80000 LINUXX86_64    1000 0x00408c41

In this sequence, we have accessed the console on pvm1 to view the clusters configuration (conf).  Next, we started the second node.  It is now displayed in the cluster's conf.

Just for fun, let's throw it the simplest of compute jobs:

pvm> spawn -4 -> /bin/hostname
4 successful
t8000b
t8000c
t4000c
t4000d
pvm>
[3:t4000d] pvm1
[3:t4000c] pvm1
[3:t4000c] EOF
[3:t4000d] EOF
[3:t8000b] pvm2
[3:t8000b] EOF
[3:t8000c] pvm2
[3:t8000c] EOF
[3] finished

There are a few things to notice about the output:

1. The command asked the cluster to spawn the command "/bin/hostname" four times.
2. The "->" option indicates we wanted the output returned to the console, which is completely abnormal... we only do this for testing.
3. The prompt returned before the output.  The assumption is that our compute jobs will take an extended period of time.
4. The responses were not displayed correctly.  They were displayed as they returned, because all this magic is happening asynchronously.
5. Each job's responses, from each node, could be grep'ed from the output using a unique serial number, automatically assigned to the job.

To leave the console and return to the command prompt issue the quit command.  All started nodes will continue to run.  To shutdown the compute cluster, execute:

echo halt | pvm

Finally, remember this one last thing: The cluster is a peer-to-peer grid.  Any node can manage any other, any node can schedule jobs, and any node can issue a halt.

Removing Old Linux Kernels

Today, I had trouble removing an obsolete kernel from my workstation. It should have been simple enough, but I tried to use yum erase rather than rpm -e, and kept running into errors. That is obviously the bad news, so let's make sure to report the good news: YUM is such an improvement over RPM alone, that it is smart enough to know which kernels are obsolete. For instance:

# rpm -qa kernel
kernel-2.6.32-279.el6.x86_64
kernel-2.6.32-279.19.1.el6.x86_64
kernel-2.6.32-279.22.1.el6.x86_64
# uname -r
2.6.32-279.22.1.el6.x86_64
# yum erase kernel
<snip>
Removing:
kernel   x86_64   2.6.32-279.el6
kernel   x86_64   2.6.32-279.19.1.el6
Is this ok [y/N]:

First, we determine the machine has three kernels. Second, we see that that it is running the most recent version, dot-22. Finally, YUM demonstrates that it is smart enough to erase the two old kernels, but not the current kernel.

One small problem: I don't want to remove dot-19 because I have a driver problem with dot-22. I only want to remove dot-null. Here's the trick:

# yum list kernel
Loaded plugins: refresh-packagekit, security
Installed Packages
kernel.x86_64   2.6.32-279.el6
kernel.x86_64   2.6.32-279.19.1.el6
kernel.x86_64   2.6.32-279.22.1.el6
# yum erase kernel-2.6.32-279.el6

The critical success factors are to drop the arch and t0 add a dash(-) between the package name and the version number.

RHEL6 Udev Rules

I recently moved my home workstation from Fedora to Scientific Linux 6, on the grounds that Fedora has diverged too far from the current RedHat distribution.  Sure, bleeding edge is cool, but as a self professed Linux mercenary, I need to be in sync with what the real world is doing... not what it might be doing.

After the move, I've found myself annoyed by the way the Gnome desktop handles removable media, in particular media cards such as Flash and Secure Digital (SD).  One trick I learned a while back, was to make sure to assign an e2label to cards formatted with an ext filesystem.  This way, when Gnome automounts the media and places an icon on the desktop, the name is the e2label.  Without an e2label, the icon's text is the device size.  This is also true of FAT devices.

The real problem, however, is the fact that the device is owned by root.  Since the desktop is running as an unprivileged user (because we never login the GUI as root... right?) we are faced with an icon for an device that we can't drop-and-drag to.  Doh! Here's how I used Udev to trick the system into allowing my GUI account to use these devices.

First, insert the device, allow it to automount, and appear on the desktop.  (We won't worry with how the kernel, udev, fuse, and the desktop is accomplishing this.)  Assuming an ext device, it was probably mounted to a dynamic mountpoint under /media; in this case, we ended up with:

# mount | grep media
/dev/sdb1 on /media/Flash_1GB type ext3
   (rw,nosuid,nodev,uhelper=udisks)
# ls -ld /media/Flash_1GB
drwxr-xr-x. 4 root root 4096 Feb  9 15:55 /media/Flash_1GB/

The goal is to modify a few mount options and change the ownership of the device.  To accomplish this, we need to tell Udev to watch for a given device and respond in a specific manner.  This requires isolating a unique aspect of the device that can be used a s trigger.  The command to manage Udev changed with RHEL6:

# udevadm info --query=all --attribute-walk --name=/dev/sdb
<snip>
looking at device '/devices/<snip>/6:0:0:0/block/sdb':
    KERNEL=="sdb"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{range}=="16"
    ATTR{ext_range}=="256"
    ATTR{removable}=="1"
    ATTR{ro}=="0"
    ATTR{size}=="2001888"
    <snip>
  looking at parent device '/devices/<snip>/6:0:0:0':
    KERNELS=="6:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    <snip>
    ATTRS{vendor}=="Generic-"
    ATTRS{model}=="Compact Flash   "
    <snip>

There are a few things to notice about this output.  On the command line, the name is the disk, not the mounted partition.  The top most block is the device, blocks that follow are upstream devices.  We are most interested in the ATTR fields.  Don't be seduced by the first directive, "KERNEL=="sdb"... we all know that Linux is notorious for changing device letters on reboot.

Second, Udev rules are created as code snippets in the /etc/udev/rules.d dir.  For simplicity sake, create a file called 99-local.rules and add all machine specific rules to this one file.  Each rule is one line.  There are many sophisticated and elegant things that can be done by Udev, but my example is a simple sledgehammer:

SUBSYSTEMS=="scsi", 
ATTRS{model}=="Compact Flash   ",
ATTR{size}=="2001888",
RUN+="/bin/sh -c 'mount /media/Flash_1GB
   -o remount,noatime,nodiratime,async;
   chown doug:doug /media/Flash_1GB' "

The first directive tells the machine that we're dealing with a disk (we could have used "block".)  The second directive is an attribute that was listed for the device (notice the spaces: it has to exactly match the output from udevadm.)  The third attribute is the device size, so this rule applies just to this card, or atleast to cards with exactly this number of sectors.  The last part of the rule is the RUN command, which executes a set of bash commands.  In this case, I'm changing the default mount options, then I'm changing the mount point ownership.  Using the RUN feature provides infinite flexibility.

dracut: FATAL: initial SELinux policy load failed

Here's an obnoxious install failure: Using the RHEL/Scientific Linux 6.3 DVD, it is possible for an install to crash on first boot with an SELinux error.  The problem is a bug in the post of the target policy RPM.  The bug is immediately fixed by running yum update... assuming you can figure out how to get the machine booted.  Luckily, the error is nice enough to tell you how move forward.

Reboot, and at the GRUB menu, append "selinux=0" to boot into Permissive mode.  From the root prompt:

ls /etc/selinux/targeted/policy/policy.24
ls: cannot access /etc/selinux/targeted/policy/policy.24:
No such file or directory

If possible, issue: yum update selinux-policy-*

If the machine is not network connected, the problem can be resolved by restoring the file policy.24 from install media.  And it it were that simple, you wouldn't need me.  You will have to force install two RPMs:

rpm -ivh --force selinux-policy-3.7.19-*
rpm -ivh --force selinux-policy-targeted-3.7.19-*

The second force install will take several minutes to complete.

Regardless of how the issue is resolved, it is best to relabel the filesystem:

touch /.autorelabel

RHEL Cluster Anti-Affinity Configuration

I'm often amused by how vendors define "High Availability", aka HA.  Customers always talk about Five Nines, but "off the shelf" HA solutions seldom achieve 99% availability.  In the case of RedHat's HA cluster service, the default configuration might provide unattended failover within 60 seconds.  Given that Five Nines only allows 25.9 seconds per month, a single failure can blow away a service level agreement.

To counter the real-world lag of application failover, an system must be load balanced across a cluster.  A real HA environment would be at least three nodes, running two instances of the service.  If one node fails, the load balancer will redirect everything to the second instance, while the service is recovered on the third node.

There is an HA problem that VMware has addressed in their HA solution, that RedHat has not, known as the anti-affinity rule.  Affinity is when two "processes" favor the same resource.  An example would be when running a web and database instance on the same machine improve performance.  In the case of redundant services, running them on the same machine is pointless, if the machine fails.  To prevent this, we need an anti-affinity rule that requires the two processes to never be on the same machine.

RedHat cluster suite provides affinity in the form of child services.  If the cluster moves the web service to another node, the database has to follow.  What they don't provide is an anti-affinity rule to prevent the load balanced services from trying to run on a single node.  As a matter of fact, by default, all services will start on the same cluster node.  (It will be the node with the lowest number.)

I found I could implement anti-affinity from with in the service,s init.d script.  First, we add an /etc/sysconfig/ file for the process, with the following variables:

CLUST_ENABLED="true"
CLUST_MYSERVICE="service:bark"
CLUST_COLLISION="service:meow service:moo"

A collision is when the presence of a service prevents this service from starting on this node.  The names should be listed exactly as they appear in clustat.  Make sure the script sources the config file:

# source sysconfig file
[ -f /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

Next, add a new subroutine to the existing init.d script:

cluster(){
  # look for other services on this host
  K=$(for J in $CLUST_COLLISION; do \
          clustat | grep "$J.*$HOSTNAME.*started" \
          >/dev/null; \
          [ $? == 0 ] && echo "$J "; \
          done)
  if [ $K ]; then
    # show service names
    echo -n "Cluster, collision $prog: $K"
    # fail, but with a success return code
    failure; echo; exit 0
  fi
  # look for this service running on other nodes
  K=$(clustat | grep "$CLUST_MYSERVICE.*started" | \
          awk '{print $2}')
  if [ $K ]; then
    # show hostname of other instance
    echo -n "Cluster, $prog exists: `echo $K | cut -d. -f1`"
    # fail but with a success return code
    failure; echo; exit 0
  fi
}

Finally, add a reference to the cluster sub in the start sub:

start(){
  if [ $(ps -C cluster-$prog.sh | grep -c $prog) == 0 ]; then
    # only check cluster status if enabled
    [ "$CLUST_ENABLED" == "true" ] && cluster
    echo -n "Starting $prog"

Here's what happens in the case of a collision:

• rgmanager issues a start
• the cluster sub recognizes the collision, but tells rgmanage that it started successfully (exit 0)
• rgmanager shows the service as running
• 30 seconds pass
• rgmanager issues a status against the service, which fails, since the init.d script lied about the service running
• the cluster orders a relocation of the service
• rgmanager issues a start... on a different node
• there is no collision this time, so the init.d runs as expected

RHEL 6 Clustering, VM Fencing

I recently retasked one of my lab machines as a RedHat virtualization server, which RedHat calls RHEV, but is really KVM.  One of this machine's tasks is to support a test cluster of VMs.  Under normal circumstances, clustering would require a remote management interface such as an ILO, DRAC, or RMM.

As usual, I was disappointed with how difficult this was.  To make matters more difficult, for you, I won't be covering clustering in the article.  This document's scope will be limited setting up to RHEV VM fencing.

On the host machine, we need to install the fence daemon.  Considering this is very lightweight, the I'm going to do a shotgun install:

yum install fence-virtd-*

On my machine, this loaded four packages: the daemon, the interface between the daemon and the hypervisor, and two "plugins".  (The serial plugin is probably not needed.)

The base RPM will provide the /etc/fence_virt.conf file.  Modify it to look like this:

listeners {
  multicast {
    family = "ipv4";
    interface = "virbr0";
    address = "225.0.0.12";
    port = "1229";
    key_file = "/etc/cluster/fence_xvm.key";
  }
}
fence_virtd {
  module_path = "/usr/lib64/fence-virt";
  backend = "libvirt";
  listener = "multicast";
}
backends {
  libvirt {
    uri = "qemu:///system";
  }
}

Two things to notice about the config file.  The key_file option is little more than a password in a text file, which is going to have to be duplicated on all the VMs in the cluster.  The "theory" is that only a device with the password will be able to fence other nodes.  This brings us to the second point, the multicast option.  If a cluster node issues a fence command, the symmetric authentication key will be multicast on the network in the clear.  Thus, the reality is that the key_file provides no real security.

Which brings us to a second issue with the multicast.  Per RedHat, cross host fencing is not supported.  As such, all cluster nodes have to exist on the same physical machine, rending real world VM clustering pretty much worthless.  Here's the reality of cross host fencing: It is not supported because of the security concerns of multicasting the clear text fencing password and the fact that RedHat cannot guarantee the multicast configuration of the switch infrastructure.  Given properly configured switches, a dedicated host NIC and virtual bridge in each host, cross host fencing works.  In this lab configuration, however, it is not a concern.

After creating a key_file, open the fenced port in IPtables:

-A INPUT -s 192.168.122.0/24 -m tcp -p tcp --dport 1229 -j ACCEPT

Copy the key_file to each clustered VM (they don't need the config file) and add the opposite IPtables rule:

-A OUTPUT -d 225.0.0.12 -m tcp -p tcp --dport 1229 -j ACCEPT

On the host, chkconfig and start fence_virtd.  Running netstat should show the host listening on 1229.  What it is listening "for" is the name of a VM to "destroy" (power off.)  This means the names of the cluster nodes and VMs recognized by KVM/QEMU have to match.  On the host, display the status of the VMs using: 

watch virsh list

Given a two node cluster, on node1 issue:

fence_node node2

On the host, the status of node2 should change from running to inactive, and a moment later, back to running.  For testing purposes, the fence_node command can be installed on the host, without the host being part of the cluster.  If you try this using yum, you'll get the entire cluster suite.  Instead, force these three RPMs:

rpm -ivh clusterlib-*.x86_64.rpm  --nodeps
rpm -ivh corosynclib-*.x86_64.rpm  --nodeps
rpm -ivh cman-*.x86_64.rpm  --nodeps

 Truthfully, the better choice is to build a VM to manage the cluster using Luci.

Kickstart from Hard Drive ISO

I'm building a machine that may need to be remotely re-imaged, without the benefit a kickstart server.  I've always heard that you can kick a machine from itself, but had never tried it.  Truthfully, it's probably more trouble than its worth.  The best option would be to install a DVD drive with media, but configure the BIOS such that DVD is after the main drive.  Since I didn't want an optical drive on this machine, here's how to kickstart a machine from a hard drive.

Let's do this backwards.  Get the machine to image off an HTTP server and then change the ks.cfg:

#url --url http://1.2.3.4/rhel6
harddrive --partition=/dev/sdb1 --dir=/

Notice that I'm telling the machine that its image is on sdb, not sda.  Providing two drives is safer that trying to image off the boot/root drive, but it could be a partition on the same drive.  Besides, I've got dozens of little drives laying around doing nothing.  Further down in the file, I also indicated:

#clearpart --all
clearpart --drives=sda

Next, we mount the second drive and copy the ISO into the root of the drive.  To clarify:

mount /dev/sdb1 /mnt
cp rhel-6-x86_64-dvd.iso /mnt/

When kickstarting from a local drive, we use the ISO file itself... not an extracted or loop mounted filesystem.  Looking back at the first change we made to the ks.cfg, we indicated --dir=/, so we are telling the installer that the ISO is on the top level of the drive.

As a matter of convenience, mount the ISO, because we need a few files from it:

mkdir /mnt/rhel6
mount rhel-6-x86_64-dvd.iso /mnt/rhel6 -o loop

Copy three files:

mkdir /mnt/images
cp /mnt/rhel6/images/install.img /mnt/images
cp /mnt/rhel6/isolinux/vmlinuz /mnt/images
cp /mnt/rhel6/isolinux/initrd.img /mnt/images

If you were going to allow the machine to rebuild different versions (or distros) you would want to add a version number to each file.

To initiate the rebuild, we will use the tried and true Grub rebuild hack:

cd /boot
cp /mnt/images/vmlinuz /boot
cp /mnt/images/initrd.img /boot
cat >> /boot/grub/grub.conf << EOF
title Rebuild
  root (hd0,0)
  kernel /vmlinuz ramdisk_size=8192 ks=hd:sdb1/ks/ks.cfg
  initrd /initrd.img
EOF

Many docs indicate that the second drive has to be made bootable.  In this case, we are still booting off the primary drive, but only long enough to read the ks.cfg from sdb and switch to install.img.  Once install.img has control, it will read the clearpart command, wipe sda, and reinstall from sdb.

There is a "gotcha" I've not quite worked out yet.  As we all know, Linux is notorious for renaming drive letters at boot time.  It is possible that  the machine might confuse sda and sdb.  This could be disastrous if the machine crashed, and while trying to rebuild, it wiped the image drive!  The good news is that the installer reports that it can't find the image and kickstart file, and fails.  Just reboot until it finds the correct drive.

* It would seem that either a UUID or LABEL could be used in both Grub and the kickstart file.  I'll add checking those possibilities to my ToDo list.  Or you could figure that part out and let me know which works.  Its only fair: I've already done the hard part for you.

Linux KVM Disk Drivers

I was having a problem with storage device names on virtual machines running on a RedHat KVM host.  Occasionally, I'd build a VM and the storage device would be named /dev/sda and other times /dev/xvda.  I quickly found that if I created the VM using virt-install, I got a xvda device, and if I used virt-manager (the GUI app), I got the sda device. After some investigation, I've discovered where things went wrong.

First, the syntax of the virt-install command changed in RHEL 6, and I was still using the RHEL 5 command.  Rather than complaining, RHEL 6 would guess what it thought I meant.  Here's the wrong command:

virt-install -n server -r 512 -w bridge=virbr0 \
-f /var/lib/libvirt/images/server.img -s 10

The -f/-s options says to create an image file the is 10GB.
Here's what was implemented:

virt-install -n server -r 512 -w bridge=virbr0 \
-disk path=/var/lib/libvirt/images/server.img,\
bus=ide,size=10

Rather than complaining that the -f/-s options were deprecated, it invoked the new syntax and assumed I wanted an IDE drive, which on RHEL 6, is named as if it were an SCSI device.  We can force the paravirt driver by using the correct command:

virt-install -n server -r 512 -w bridge=virbr0 \
-disk path=/var/lib/libvirt/images/server.img,\
bus=virtio,size=10

Second, the GUI does not allow a VM's disk type to be selected from the install wizard-- it always defaults to the paravirt driver.  To force a specific driver, on the last screen of the wizard, check the box for "Customize configuration before install".

This will open a new window listing the VM's hardware.  Select "Add Hardware", select "storage", and configure a second disk the same size as the first.  At this point, there is a pull down menu that will specify the driver.  Once the second disk is in place, remove the first.  Removing the first disk before the adding the replacement disk can cause problems.

Hint: Once you've modified the hardware, there is not "Apply" option.  Just close the window and the VM will launch.

Tracking SSH Tunnels

Native to Secure Shell (SSH) is the ability to create point-to-point, encrypted, tunnels.  The function was designed to provide legacy protocols, such as mail (SMTP/POP) with encryption.  A user could login to an SSH server in their company's DMZ, open a tunnel from their laptop to the server, and redirect their mail client through the tunnel.  On the surface, this sounds like a good idea: it protects the exchange of company data from the "protected" corporate intranet to users "in the field".

But, as with all good things, there is room for abuse.  Consider the opposite scenario:  What if a user inside the corporate intranet SSH'ed to the DMZ server and built a tunnel to allow them to surf the web, thus bypassing the content filters?

Granted, content filters are just a way for the man to oppress middle class workers.  By censoring free thought, the 1% is able to keep the 47% running on the hamster wheel of consumerism.  Hear me, my brothers!  There will come a day when the proletariat will raise up and declare their freedom from the jack-booted thugs of Wall Street and their Illuminati masters.

But I digress...  Where was I?  Oh yes, SSH tunnels. So the question is this:

How can we monitor the SSH tunnels defined on the server to ensure they are not being abused?

Much to my surprise, the answer is:  You can't.

There does not seem to be any mechanism for determining what tunnels exist, and here's why.  The tunnel is defined on the client end, where the SSH application is always listening.  When the client receives a packet that matches a tunnel, the packet is shipped to the server with handling instructions.  When the server gets the packet, it opens the needed socket, fires it off, then closes the socket.  In other words, the connection from the server to the destination is not persistent... it behaves more like UDP than TCP.

Since a socket is opened, it is possible to capture it with lsof -i, but since the socket is transient, trying to catch it in a while/do loop is a matter of pure luck.

This means we have two choices, one of which shouldn't count.

In order to catch someone using a tunnel to surf out of the DMZ, we need an IPtables rule to catch the outbound packets.  As it turns out, any packet originating from a tunnel will use the server's IP address as the source address.  We only need to log the initial connect, so we only need to log the SYN flag.  To further complicate things, our abusive user has to be using a proxy, so we can't restrict our checks on port 80 and 443.

iptables -A OUTPUT -s 192.168.0.1
  -o eth0 -p tcp --syn
  -j LOG --log-prefix "out-syn "

Here, we are looking for OUTPUT, since we are assuming that this DMZ machine is supposed to be building tunnels.  The (-s) address is the address of the DMZ machine.  In this case (-o) eth0 is the internet side of the machine and eth1 would be the intranet side of the machine.  Notice that no port number is assigned to the (-p) TCP statement.  Lastly, we are going to log this message.  (The trailing space in the quotes is significant.)

This rule will catch bad tunnels, but ignore good tunnels, on the grounds that good tunnels will use (-o) eth1 to get to the intranet resources.

If you'll recall, I said there were two choices.  The second is this:

iptables -A OUTPUT -s 192.168.0.1
-o eth0 -p tcp --syn
-j DROP

In this case, we are refusing all outbound TCP traffic from the DMZ machine.  (Since DNS is UDP, we can still resolve the addresses of the inbound SSH connections.)  As stated above, we are allowing the good tunnels, since they use (-o) eth1.

So which of the two rules shouldn't matter?  The first:  We shouldn't have to "catch" abusive users, we should just stop them.  Of course, we could use both lines to first log them and, second, prevent the connection.  This allows us to know who the abusers are, and bitch slap them for their feeble attempt-- for they are probably using Windows workstations, and deserve to be degraded.

What's that you say Mr. Boss?  You want me to prove abuse exists before locking down the DMZ.  Okay, we implement rule number 1, log the abuse, and then later lock down with rule number 2.

What's that you say Mr. Boss?  Prove the abuse exists without implementing rule number 1.  Ah...  No can do.

Oh well, if you want me, I'll be in my cube.  Listening to Slacker internet radio, via an SSH tunnel, through the DMZ.

Use awk To grep "this but not that"

I've run into this situation several times over the decades, bit for some reason I never researched an elegant solution. Consider the case of grep'ing to see if a process is running. The simple solution is:

ps -ef | grep "ntpd" 

The problem is that if there is one process matching the regex, this will report two processes, because it will also report the grep process that is grep'ing the process stack. Its kind of like taking a picture of yourself in the mirror. The generic solution to this is:

ps -ef | grep "ntpd" | grep -v "grep" 

In other words, lets launch another grep to grep out the grep. This is only slightly less efficient than taking a picture of yourself in the mirror, then Photoshopping the camera out of the picture.

Today, I found the elegant solution, and its... awk to the resuce!

ps -ef | awk '/ntpd/ && !/awk/' 

Here, awk is taking the stream and searching for a line that has ntp and (&&) does not (!) have awk.

Good-bye Gnome

In the latest machine re-org, I reloaded my workstation with Fedora 15. I don't like it. I think they've made several wrong turns, the single biggest being their implementation of Gnome 3.

With Gnome 3, they have set a minimum acceptable video hardware platform. That's fine, because they provided a fallback mode in case the machine does not support 3D rendering. Accept for one minor problem... most of the desktop's features do not work in fallback mode.

For instance... The ability to exit the GUI. Yes: there is no logout, quit, exit, stop, leave, or otherwise feature. The only way out is to open a terminal and init 3. If they missed something as important as an exit function, just image what else they missed. (Hint: allot!)

So, I'm trying to switch to KDE and it looks promising. As a matter of fact, I have found a fix for my single biggest complaint with KDE. I am so use to highlighting text in a terminal window and pressing "Shift-Insert", that for five years, I have refused to use KDE because it required the extra step of "Ctrl-Insert".

On the taskbar, near the clock is a scissors icon. Left click, select Configure Klipper, and click "Synchronize contents of the clipboard and the selection". Now it works the way I want!

Linux Rescue for VM on XenServer

I finally figured out how to rescue a Linux VM running on a Citrix XenServer host, an believe it or not, it is completely unintuitive! First, power off the VM, though if you're going to rescue mode... you're probably "down" already. Second, mount the rescue media (CD/DVD). Third, make sure the VM is highlighted on the left pane of XenCenter.

Here's the trick: Across the top of the XenCenter menu, select VM, and Start/Shut down, and Start in Recovery Mode. The machine should boot from the rescue media. Proceed per rescue SOP.

Be forewarned... For some reason, booting is incredibly slow.

SSH Tunneliung of X11 Apps

I had occasion to finally test something I've been wondering for a while: What is the minimal configuration to allow and X11 application to tunnel through SSH? First, this procedure assumes you have a workstation that can display X11 applications. This can either be a Linux desktop or a Windows machine running Xming or another lesser X client.

Second, install a system with only the core or base packages, possibly by building the system through kickstart. Third, edit the /etc/ssh/sshd_config and make sure X11Forwarding is set to yes. Reload if needed.

Next install a simple graphical application; for my test I used xclock:

yum install -y xorg-x11-apps

Attempt to run:

ssh 192.168.1.1 -X xclock
Error: Can't open display

In this case a failure is what we expect.

Conventional wisdom says we need to install the entire "X11 Window System" group, which will grab almost 100 packages. Instead install one RPM:

yum install -y xorg-x11-xauth
ssh 192.168.1.1 -X xclock
Warning: <snip> (repeated several times)

...but behold! A glorious xclock. The errors are from not loading fonts on the remote machine. Oddly, if we install xterm, it also complains, yet it works just fine.

A side note on this procedure: To prevent from issuing the -X (or -Y) with the SSH command line, change /etc/ssh/ssh_config, adding:

ForwardX11 yes

Adding gedit will require 57 more packages.
Adding kedit will require 68 more packages.
Best choice: gvim, requiring three packages.

yum install -y xorg-x11-fonts-Type1 xorg-x11-fonts-misc
yum install -y gvim

Changing Linux/Unix "ls" Time Format

I though I had documented this, but had to look it up again today. To change the format of the the ls command's time, use:

ls -l --time-style=+%s

...where the format is the same as those listed for the date command. See man date. I like +%s, because it allows scripts to calculate "elapsed time" between two file modifications.

Another useful option --time to display atime or ctime rather than mtime.

RHEL 6 Virtualization, Memory

There are two memory settings presented in virt-manager: Allocation and Maximum. The Allocation setting is what will appear in /proc/meminfo and top. The Maximum is what can be used to boot the VM. Interestingly, the kernel will refuse to boot with out enough RAM, but once booted, will run with significantly less.

The Maximum value needs to be more than 348MB. Any lower, and boot time is noticeably slower due to swap activity.

The Allocation value needs to be more than 148MB. The VM won't crash until the allocation is about 115MB, but there are several factors that could effect that number. Obviously, 148MB may not meet a every VMs needs, but it seems to be the lowest reasonable limit.

Keep in mind, the Maximum is always allocated to the VM at boot time, and them lowers to the Allocations, so don't set it too high. The Allocation can be dynamically changed for a running VM, but cannot exceed the Maximum. Any changes to the Maximum require a reboot.

RHEL 6 Virtualization, Paravirtualization

Paravirtualization is a big deal. It is avoided by VMware, is alchemy in Citrix Xen, and is cryptically alluded to in RHEL 6. Yet, for those of us that are almost exclusively Linux, the performance and density advantages are huge. Even Windows XP performance is noticeably improved running "para-virt". (As for Vista and Win7... they're both hogs, no matter what.)

In ten words or less, paravirtualization improves performance by loading a version of the operating system optimized for the host's hypervisor.

In the RHEL 6 Virtualization guide Chapter 8, they state that para-virt does not work with KVM. This would imply that there is no way to optimize a RHEL 6 VM on the RHEL 6 platform. Given that, why not just run VMware?

Yet in Chapter 11, they mention that para-virt drivers are automatically loaded and installed for RHEL 6 VMs and Linux VMs based on the 2.6.27 or newer kernel.

So... Which is it... Para-virt yes or para-virt no?

Survey says: Kernel no, drivers yes. But, of course, there's a catch.

Once the VM is installed and running, execute an lsmod | grep virtio. Look at the last line. The items at the right of the numbers will indicate which para-virt drivers are used. You want four of them, but may only have three. Depending on how the VM accesses the outside network, the virtio_net driver may be missing.

To enable virtio_net, a specific sequence of events must be followed:

1. Power off the VM
2. From virt-manager, Open the VM
3. Select View and Details
4. Select NIC
5. Change Device model to "vrtio"
6. Apply, exit, and Run the VM

Upon boot, the virtio_net driver should be listed.

Call it a bug, but if you try this with the VM powered on, it will claim to work, but will not.

As for performance without a para-virt kernel, I am still a little skeptical.

RHEL 6 Virtualization - Bridged Interface

With version 6, Red Hat has finally fixed the long standing problem of not being able to use the GUI to configure a Virtual Machine shared connection. Lets first review the types of VM network connections:

* RH=y  CX=y  VW=y Internal
* RH=y  CX=y  VW=y Dedicated or Slaved NIC
* RH=y  CX=y  VW=y Routed or VLAN'd
* RH=y  CX=y  VM=y Shared or Bridged
* RH=y  CX=n  VW=n Network Address Translation (NAT)
(Bold entries are the "default" config.)

An internal connection does not route traffic off the host system. A dedicated (also called slaved) connection, requires a separate NIC for each VM, which is very inefficient. A connection that is routed or VLAN'd requires the network be aware of the the specialized configuration. A shared or bridged connection (what we're after) extends the real world subnets in to the virtual machines. The last type, NAT, allows the VM's to communicate out, but does not permit inbound request, thus rendering it useless to servers.

Unfortunately, Red Hat uses NAT, by default, and their virtualization technology is principally used for server consolidation. As if this is a good idea. To make matters worse, their documentation still suggests manually configuring a shared connection, and does not explain that once you're done, you won't be able to see the connection in the GUI.

Here's how to do it right:

From virt-manager, connect to the host, click Edit, and select Host Details. On the Network Interfaces tab, click the plus sign (+) at the lower left. In the pop-up, select "Bridge" and Forward. Assign a name-- I recommend br followed by the eth number of the card you are sharing. In other words, if you are sharing eth1, name it br1.

On the same screen, set the start mode to "onboot", check "Activate now", and check the target NIC that you want the VMs to access. Take a deep breath, hold it, and click Finish. Scary things will happen, but after about a minute, the window should respond.

Notice that the eth item has disappeared from th elist and been replaced by the newly defined bridge. Now click the Virtual Networks tab, and notice nothing has changed. Why doesn't the new connection appear in the list? More evidence that Red Hat's interface is the least intuitive of all the vendors. This tab is a list of virtual networks, and a bridged connection is an extension of the physical network. (Yeah, while technically correct, it doesn't make sense to me either.)

When provisioning VMs, make sure to select expand Advanced options and choose the br device.

*** NOTE ***
An excellent discussion of the underlying technology is available on Dale Bewley's blog. You'll need this for kickstarting hosts.

RHEL 6 Virtualization As Non-root User

Red Hat has always been overconfident about the use of the root account over SSH. By default they allow direct logon to the root account via SSH, because the first "S" stands for Secure. But that's not the point... You should never let anyone logon directly as root. Always access as a user, and escalate privileges.

With RHEL 6 virtualization, I ran into a problem with virt-manager, in that it refused access from a non-root account. If I logged in as myself, the app would start, but not allow connections, complaining:

Unable to open a connection to the libvirt management daemon.
Verify that:
- The 'libvirtd' daemon has been started

Further digging (clicking "Details") shows a couple Permission denied messages. Using sudo doesn't help, switching to root doesn't help, but SSH'ing in as root works. So, what's the fix?

Turns out there is security group that allows the identification of trusted users. Edit the /etc/libvirt/libvirtd.conf file and uncomment three lines:

unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"
auth_unix_rw = "none"

The third line bypasses the "polkit" security mechanism... since it doesn't work, anyway.

The natural reaction would now be to issue a service libvirtd reload command, but not so fast... that won't work. (But feel free to try it if you like-- I'll wait.)

List the /var/run/libvirt dir. You should see a libvirt-sock file owned by root.root with permissions of 700. Use service libvirtd stop, and file will disappear. Issue service libvirtd start and it should fail to start, claiming it can not load the configuration file, which is not true. The problem is that we told it to allow access to members of group "libvirt", which does not exist. Create it, add yourself:

groupadd -g 170 libvirt; usermod -a -G libvirt doug

Logoff, login, and verify with the id command.

Issue the start command again, and the service should start. List the /var/run/libvirt dir. This time, the file is owned by root.libvirt with permissions of 770. We can now access and control the server as a non-root user.

RHEL 6 - Network Gotcha

Earlier, I had reported problems with my Atheros integrated NIC on my new mainboard. I finished the build on my RHEL 6 cloud node by adding an Intel Pro100 I had lying around as eth1. As expected, Anaconda recognized the card on install. To my surprise, the card was not active on first boot. But then the real surprise... I concurrently installed a RHEL 6 VM, and its virtual 8139 was also not energized on boot.

Something wicked is afoot, and it smells putridously similar to:

NetworkManager

When we cat ifcfg-eth0, by default, the file includes DEVICE, HWADDR, ONBOOT equal "no" (the symptom), and NM_CONTROLLED equal "yes" (the disease). I've hated NetworkManger ever since it showed up on RHEL 5. (It uses that ridiculous mixed cap, no space, spelling because the Red Hat developer who "innovated" this solution wishes he worked at Apple instead, and that's the way Apple processes are named.) My traditional solution was to chkconfig NetworkManger off, hardcode the network params, and life was good.

There is a perplexing part to this story, however, that forces more review. On RHEL 6, if you select @Base, @Minimal, or @Server-Platform, NetworkManager is not installed. That's fine with me, but the cards are tagged as being controlled by a service that doesn't exist. So, I approached this from the point of view that just as we are going to have to learn to use postfix instead of sendmail, know how to configure static networking with NetworkManager. Preferably, without the GUI.

So, I installed a slew of RPMs from DVD to get the NetworkManager service running. I read man pages, researched in the web, played with the tools, and found the solution. The solution is to hard code the static IP in a script under /etc/NetworkManager/dispatcher.d. But wait a second... If we have to hard code the data in a script, why not put it in /etc/sysconfig/network-scripts as we have been for over ten years. Good idea: Let's uninstall all this piece of crap.

The workarounds:
* Kickstart the server with a static address
* Set the address using firstboot
* Manually edit the interface files
* Use system-config-network-tui to set the address. (Don't try to use the TUI to set it to DHCP, it won't work)