Squid Reverse Proxy Server

I have a small problem for which a Squid reverse proxy server was the obvious problem, but I had the hardest time getting this to work as advertised. Why was it difficult? Undocumented options.

Here's the problem: I've got three webservers with identical content. I want eveybody from comcast.net to hit one, verizon.net to hit the second, and everybody else to hit the third. Squid can be configured to grab the requests, do the lookup on the domain, and forward them to the correct server. Here's a snippet of the configuration that worked:

http_port 80 accel defaultsite=gfs2.terran.lan
cache_dir null /null

acl first  src comcast.net
acl second src verizon.net
acl others src 0.0.0.0/0


cache_peer 192.168.69.59 parent 80 0 proxy-only \
  no-query no-digest originserver name=gfs3
cache_peer_access gfs3 allow first
http_access allow first

cache_peer 192.168.69.60 parent 80 0 proxy-only \
  no-query no-digest originserver name=gfs4
cache_peer_access gfs4 allow second
http_access allow second

cache_peer 192.168.69.61 parent 80 0 proxy-only \
  no-query no-digest originserver name=gfs2
cache_peer_access gfs3 allow others
http_access allow others

Yummy, Delicious, tar

Create an archive, based on date. The man page says use:

-N, --after-date DATE, --newer DATE

But what's the format for date? Further, if you try:

tar -czvf --newer "11 Mar 2009" www-incr-0311.tar.gz \
    www

...it creates a bogus file, --newer, which is a bear to delete.

(Try it. Frustration is good for the soul. The trick is to use the absolute path to file.)

Turns out, with tar, the dash-dash options must go at the end of the command line:

tar czvf www-incr-0311.tar.gz www \
    --newer "11 Mar 2009"

As for the date format, I decided to use mtime and an all numeric format:

tar -czvf www-incr-0311.tar.gz www \
    --newer-mtime "20090301"

Updating XenServer

I'm a little disappointed that the XenCenter management platform for XenServer only runs under Windows, but then VMware VIC only runs under Windows, so I shouldn't really expect much from them. What's more odd is the way the XenServer is updated. Since the box is a bastardized version of RHEL 5, I expect Yum to be the update mechanism. Here's how it works, instead:

1. XenCenter polls Citrix for updates and provides a list.
2. Items are downloaded via an IE download manager plug-in to the Windows station.
    (Select "Save", not "Install"!!!)
3. Navigate to the download folder, double click the xsupdate file.
4. XenCenter will launch with a system update wizard.
5. Select the servers to be updated.
6. Sit back and watch all the pretty colors flash by.

Just don't like having to use Windows to update Linux.

XenServer Licensing

Since I have nothing better to do with my time, I've been playing with the Citrix XenServer and XenCenter. I imagine this will change after the March 25th update, but I noticed that XenCenter only applied the license to the first server you brought online. My assumption was that it was replicated to all others as they were added, but that does not seem to be the case.

To license a new server, highlight the machine in the inventory. On the menu bar, click Server, and select Install License Key. Should be point an click from there.

Linux iSCSI Strangeness

To connect to an iSCSI LUN using the Linux iscsiadm command, we issue two commands. It turns out the syntax of the second command is completely unforgiving. The two commands that do not work:

# iscsiadm -m discovery -t sendtargets -p localhost
127.0.0.1:3260,1 iqn.2009-02.lan.terran:lnxadm3.cumulus
# iscsiadm -m node --login -T iqn.2009-02.gov.gao:lnxadm3.cumulus -p localhost:3260
iscsiadm: no records found!

Notice it failed. The problem is that the part of the second command that says localhost:3260 absolutely must say what was reported by the discovery. In other words, it must say 127.0.0.1:3260. When this is run across a network, we see the same behavior. The correct syntax:

# iscsiadm -m discovery -t sendtargets -p lnxadm3.terran.lan
192.168.93.37:3260,1 iqn.2009-02.lan.terran:lnxadm3.cumulus
# iscsiadm -m node --login -T iqn.2009-02.lan.terran:lnxadm3.cumulus -p 192.168.93.37:3260
Logging in to [iface: default, target: iqn.2009-02.lan.terran:lnxadm3.cumulus, portal: 192.168.93.37,3260]
Login to [iface: default, target: iqn.2009-02.lan.terran:lnxadm3.cumulus, portal: 192.168.93.37,3260]: successful

Snow in DC

A couple weeks late, but a respectable storm. When my supervisor said wondered into work around 10, he said "I can't believe you came to work in this snow."

I pointed out that snow didn't stop Patton, so I wasn't going to let it stop me.

Updated cluster.conf Format

I'm faced with having to R&D a cluster on my own equipment, which I'm proud to say is rather up to date, and take the configuration and migrate it into a customer site that is, well, not so current. Another issue I have to deal with is ensuring that the tools they have can maintain the cluster going forward. Thus, it has to be as generic as possible. Here's an updated version of cluster.conf which seems to work with GFS and the sys-con cluster tool:

<?xml version="1.0" ?>
<cluster config_version="3" name="clust">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
  <clusternode name="gfs1" nodeid="1" votes="1">
    <fence/>
  </clusternode>
  <clusternode name="gfs2" nodeid="2" votes="1">
    <fence/>
  </clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
  <fencedevice agent="fence_xvm" name="fence_vmx"/>
</fencedevices>
<rm>
  <failoverdomains/>
  <resources/>
</rm>
</cluster>

The big difference is the format of the fence lines. From a pure XML standpoint, it is a strange config.

Activating an Inactive Raid Element

You know its a tough problem when even the title needs an explanation. Consider a two disk, mirrored, software raid. If one disk fails, our data is safe. Imagine that disk A has the OS and a data partition. Disk B is the mirror for disk A's data partition. Now, disk A fails. How do we access the data on disk B (the mirror) on another machine... Without damaging the data!

Obvious first step, get it to a second machine. Boot the box and ensure the disk appears in fdisk. All good from a hardware standpoint. Lets look at the software:

# cat /proc/mdstat
Personalities :
md_d0 : inactive sdb1[0](S)
5237056 blocks

unused devices:

Since our partition is a type fd, it was recognized, but the kernel didn't have enough info to reactivate the raid.

Without taking you though the failures... well, just one:

# mdadm --assemble -v /dev/md0 /dev/sdb1
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has no superblock - assembly aborted

(That's to seed the errors into the search engines.)

As I was saying: Without taking you through all the failures, here's how we get to our data.

# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
# cat /proc/mdstat
Personalities :
unused devices:
# mdadm --assemble -v /dev/md0 /dev/sdb1 --run
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: /dev/md0 has been started with 1 drive (out of 2).
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0]
    5237056 blocks [2/1] [U_]

unused devices:
# mount /dev/md0 /mnt
# ls -l /mnt
total 16
-rw-r--r-- 1 root root     0 2009-02-26 23:38 keep.me

Making GFS available to Nodes

A big misunderstanding with GFS is that it is a network aware filesystem. Actually, you still need something to get the date from the server to the clients. For RHEL5, that something is GNBD... For anything newer than FC4... Its something else?

Assuming you have a two node RHEL 5 cluster, with a GFS file system presented to one node, we will now make it visible to a second node. First, load the needed packages:

yum install gnbd_utils

Turns out this process is not as well automated as most services. My guess is that this is a temporary solution, most likely to be replaced with iSCSI.

On the server side (side with the GFS volume), append to /etc/rc.d/rc.local:

gndb_serv
gndb_export -t /dev/gfs/test -e shared

Make sure the volume name on the server is unique and not VolGoup00 or vg0.

On the client side, append to /etc/rc.d/rc.local:

modprobe gndb
gndb_import -i gfs1.terran.lan

In this example, we exported an LVM, so we'll do a vgsan to see it.

Single Node GFS

Pretty much just for testing, here's a single node cluster.conf:

<cluster name="clust" config_version="1">
  <clusternodes>
    <clusternode name="gfs1.terran.lan" votes="1" nodeid="1">
      <fence>
        <method name="single">
          <device name="fence_dev" nodeid="gfs1.terran.lan"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <fence_devices>
    <fence_device agent="fence_xvm" name="fence_dev"/>
  </fence_devices>
</cluster>