An EC2 Conundrum

Whenever I would lecture on Amazon's EC2, I would point out that Amazon's internal infrastructure is (effectively) EC1, and when they need capacity, it comes from EC2. In the past few weeks, I've seen this first hand. Of course, we need to remember that this is Amazon's peak period, so a resource crunch should be expected as part of the normal patterns of business activity (BPA), but I was caught by surprise in this one respect.

The Amazon cloud, known as Amazon Web Services (AWS), is billed on three meters:
* VM Resources, such as CPU and memory
* Storage, either block volumes or web based files
* Bandwidth, both into and out of the cloud

I have a set of VMs that I launch as needed, so I am not always billed for cycles or bandwidth. When I need the VM, I don't want to have to upload all the supporting applications and data, or go through a complex configuration procedure. The solution was to grab an Elastic Block Store (EBS), which looks to a Linux VM as a disk device. I provision the VM, connect the volume, log in, mount the device, where I have a set of scripts that rebuild the application server in less than 1 minute.

Here's where I got burned: The EBS is actually a LUN on a SAN, which resides in a data center, somewhere in the world. Amazon has four regions: Virginia, California, Ireland, and Singapore. I picked Virginia. But in Virginia, they have four data centers, called availability zones, labeled A, B, C, and D. My volume is in Virgina "B". Unfortunately, they have insufficient capacity in VA-B to launch a VM, as of about 21 December.

This means I've got stuff on a disk, somewhere across the Potomac, that I can't get to, because I don't have a machine to access it. I could launch a VM in VA-C or VA-D, but there is no native mechanism to allow VMs to mount disks that live in another data center. Thus the conundrum: How do we protect against this situation?

The answer is obvious: clustered replication. Two EBS volumes in different data centers, with one VM acting as the master node, and another VM acting as the replication node. Unfortunately, this doubles the cost of the system... From $15 a month to $30 a month. Not really that much... and that assumes my data is critically important, which it isn't.

But you'd think Amazon would have provided a way to prevent this from happening. After all, its not like me paying twice as much on a monthly basis is something they'd actually want to happen.

A Second Apache Instance With YUM

I ran into a situation where I needed two separate instances of the Apache HTTPD service on the same server. I couldn't simply virtual host the second site, because it needed a radically different configuration from the first instance. My first reaction was that I would use YUM to install the first instance, and then snag the Apache source and compile the second instance. The problem with this was that the two instances would be different versions: the first easily patched and upgraded with YUM and the second being an administrative nightmare of patching and recompiling.

To prevent the overhead of administering the second instance, I started investigating an old RPM option I'd never used: --relocate. It turns out this option was the opposite of what I had expected, in that it moved the first instance, rather than install a second. And besides, using RPM manually was only incrementally better than the original idea.

So what about YUM? There is an option for --installroot=/path. Seems like what I wanted: instead of distributing files based on system root to /etc, /usr, /var, lets put the httpd files in an tree under /opt. What happened when I ran the command surprised me:

yum install httpd --installroot=/opt
<snip>
Transaction Summary
=============================
Install 77 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 64 M
Is this ok [y/N]:

This thing is not going to only install the httpd binaries, libraries, and config files... but every dependency... which already exists on the system! And its going to require 64M of disk space!

Oh, wait... I've got like 250G of free space. Do I really care if it take 64M? No! And so, I answered "Y". What did we end up with?

ls /opt
bin dev home lib64 mnt proc sbin srv tmp var
boot etc lib media opt root selinux sys usr

Ouch! That's ugly. Looks like the better (cleaner, prettier) choice would have been:

yum install httpd --installroot=/opt/httpd-2i

For the sake of simplicity, through the miracle of virtualization, lets just consider that fixed.

Lets see what we got:

/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)

What about an update? I added a repo file to include the updates directory on the satellite server.

yum update -y httpd
<snip>
/usr/sbin/httpd -v
Server version: Apache/2.2.9 (Unix)
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)

As expected for the base install, but no love from the second instance.

yum update -y httpd --installroot=/opt/httpd2i/
Setting up Update Process
No Packages marked for Update

Still no good. As a matter of fact, nothing seemed to work. So, as a workaround, I tried this:

yum install httpd -y --installroot=/opt/httpd2i-2/
rsync -Pr httpd2i-2/* httpd2i/ --update
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.9 (Unix)
rm -rf /opt/httpd2i-2

In a nutshell, create a third instance, and copy the third instance over the second, hoping not to overwrite any configuration files in the process.

Does this solve the original problem? Sort of. Is it easier than recompiles? Its faster. Just one more problem... As is, the new Apache does not run. Looks like we need some more hacking. Stay tuned for part 2.

*** Update ***
On second though... I'll just recompile. It turns out there are some references to that path in the RedHat binaries. That's bad form on their part, and they should be ashamed, but by the time I figure out how to hack this, the recompile will be done.

So, no part two. Just snag the binaries and be done with it. That doesn't mean that this feature is useless. It just means that it didn't solve this problem.

Evil Free Cell Game: #29868

I finally beat it. Now I can go to bed.

So, So, Sad: And its ITIL's Fault

I signed up for a series of ITIL classes with the goal of earning the ITIL Expert Certification. (They're choice of words, not mine; I'm leery of "experts", personally.) There are five classes, each with a test. Once you pass all five classes and tests, there is a sixth class and test.

Yesterday, I found out that I failed the fifth test. I was crushed! Not because I failed: I fail all the time. Constantly. As a matter of fact, I failed tests one and two, but those didn't bother me. Let me explain.

I call the testing format Three Little Kittens.

You get a case study.

Three little kittens, have lost their mittens.

You must select "the best" solution, based upon four chioces:

1. They bought gloves
2. They found their mittens
3. And they shall have no pie
4. Kittens don't wear mittens

The operative factor in this process is the fact that we have to pick "the best" solution. The answers are weighted with scores of 5, 3, 1, and 0 points. In the case study above, the answers logically break out as follows.

1. Throw money at the problem
2. A definitive solution
3. Punishment does not solve the problem
4. True, but irrelevant

Thus, the 5 point answer is "2", the 3 point answer is "1", the 1 point answer is "3", and number "4" is worth nothing, even though it is completely accurate.

I failed the first two tests because I did not personally recognize the level of dedication that is needed for the certification track. Furthermore, the class vendor, Global Knowledge, has not done a good job of setting expectations. Embarking on this process requires either significant management and project experience, or the purchase of supplemental material and several weeks of study before the class.

This certification also requires complete support from your employer. They have got to be willing to give you the time and resources to succeed. They have got to recognize the value they will receive from this process.

I have scheduled to retake the class and tests for 1 and 2. After passing tests 3 and 4, I was very confident that I understood the testing method, and the amount of preparation needed before hand. My results for test 5?

50% of answers were 5 pointers
12% of answers were 3 pointers
 0% of answers were 1 pointers
38% of answers were 0 pointers

Fail!

So, whose fault is this? Doesn't matter (see justification 3 above.) But I am sad.

Why Am I Changing Light Bulbs?

Being an ecologically conscious kinda guy, about three years ago, I went through my house and replaced my incandescent bulbs with CF bulbs. It was hugely expensive because I bought good GE brand bulbs. I saw it as an investment. Not only would the bulbs save energy, but they would last a thousand years.

Yes, a thousand years, damn it. They said the bulbs lasted five times longer than "normal" bulbs. They said that even though they cost more, they don't really cost more when you consider the cost of all the bulbs you won't have to buy in the future. And you're saving energy.

Yeah, because they put out half the light of normal bulbs. One bathroom had a two bulb fixture. If you walked into the bathroom in the middle of the night, it took five minutes for the lights to power-up. (I was already "done" by then.) So, I changed one of the CF bulbs for a normal bulb. For short jobs, the incandescent bulb fires up giving us 50% light, and for jobs taking more than 15 minutes, the CF gets us up to 90%. And still saves energy.

But the CF bulb has burned out. Not the normal bulb. The CF that was suppose to last five times longer! What's the deal? It's almost like they lied to me... or something.

Oh now: lets not loose sight of what's important. I'm saving the planet. I'm being environmentally aware. The CF bulbs reduce my carbon footprint. And they contain deadly levels of mercury that is sufficiently toxic that improper disposal is criminal in some jurisdictions.

Now if you'll excuse me, I've got some endangered tigers that need to be shot.

Oops, I've Seen All of Netflix

Yep. All of it.

Okay, not every movie in the entire Netflix inventory, but every movie that I've ever wanted to see. I'm down to watching foreign flix with subtitles. I'm to the point that I'm saying to myself: "Hey, I don't think that was all that bad. Oh sure, I purposely avoided it when it came out at the theaters since it wan't worth spending money on, and I didn't watch it on HBO or TV since I had better stuff to do, but I'll add it to my Netflix queue."

Why? I because Netflix is a flat rate service. Its a buffet. All you can eat. There was an old Huey Lewis song:

"The sign on the door said all you can eat for $1.99, but one dollars worth was all i that could stand"

(Like there is anything but old Huey Lewis songs.

So here's what I have left:
- The Good, the Bad, the Weird: A subtitled Chinese Western
- How to Train Your Dragon: What passes for a cartoon these days
- Daybreakers: Because the world needed another vampire movie
- Scott Pilgram vs The World: For the 80's video game references

Sometime in the future, they'll have:
- Despicable Me: I dream of world domination
- Inception: Speaking of dreams
- Skyline: Gotta keep an eye on the aliens

I hope somebody invents something Earth shatteringly entertaining... quick.
*Sigh*

Fedora and Screen Resolution: Pt2

Turns out, the xrandr info is not persistent across reboots. The obvious solution was to add the needed lines to the /etc/rc.local, but xrandr will only execute if the X engine is online. The solution was to create a script and drop into the X window systems initialization sequence:

cat /etc/X11/xinit/xinitrc.d/setres.sh
#!/bin/sh
xrandr --newmode "1280x1024" 108.88 1280 1360 1496 1712 1024 1025 1028 1060 -HSync +Vsync
xrandr --addmode VGA-0 "1280x1024"
xrandr -s "1280x1024"

Notice the scripts path.

Fedora and Screen Resolution

For the last few revisions of Fedora, I've had problems with getting my physical systems to behave at the desired screen resolution. The problem is the way the video drivers attempt to detect them monitor-- it is assumed that the display hardware is an LCD. Unfortunately, some of my physical systems still use CRT's. Why? Because they work.

I don't know exactly when the change occurred, but starting with F13, my system always booted to 1024x768. Back in the day, we'd use system-config-display to set the proper resolution, but it is no longer a default component, and has become unreliable. Instead, we should use the less intuitive xrandr. So, here's what it takes to override the default resolution.

First, determine what xrandr sees. From within an X desktop environment, open a terminal window.

xrandr -q
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2944 x 1024
VGA-0 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0*
  800x600       60.3
  640x480       59.9

It knows that Screen 0 (the monitor) is capable of 2944x1024. For some reason, VGA-0 (the video card) is defaulting to 1024x768 as indicated by the asterisk.

Second, we need to configure more video modes. This requires that we feed xrandr a huge amount of information we don't have. Luckily, gtf knows what we need. Let's try to bump the resolution up one notch to 1152x864 with 60Hz refresh rate.

# gtf 1152 864 60
# 1152x864 @ 60.00 Hz (GTF) hsync: 53.70 kHz; pclk: 81.62 MHz
Modeline "1152x864_60.00" 81.62 1152 1216 1336 1520 864 865 868 895 -HSync +Vsync

We need the second line to feed to xrandr:

xrandr --newmode "1152x864_60.00" \
  81.62 1152 1216 1336 1520 864 865 868 895 -HSync +Vsync
xrandr -q
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2944 x 1024
VGA-0 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0*
  800x600       60.3
  640x480       59.9
  1152x864_60.00 (0x7e)   81.6MHz
    h: width 1152 start 1216 end 1336 total 1520 skew 0 clock 53.7KHz
    v: height 864 start 865 end 868 total 895 clock 60.0Hz

Our new mode has been staged, now it has to be connected to a video card, and activated:

xrandr --addmode VGA-0 "1152x864_60.00"
xrandr -q
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2944 x 1024
VGA-0 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0*
  800x600       60.3
  640x480       59.9
  1152x864_60.00   60.0
xrandr -s "1152x864_60.00"
xrandr -q
Screen 0: minimum 320 x 200, current 1152 x 864, maximum 2944 x 1024
VGA-0 connected 1152x864+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0
  800x600       60.3
  640x480       59.9
  1152x864_60.00   60.0*

If you're lucky, you can still see your video output!

Oddly, it would seem the part in quotes is completely arbitrary, but GNOME's desktop resolution utility needs that field to be in the 1234x567 format. The real headache is trying to determine the available resolutions and scan rates. If you really feel daring you can pump all the possibilities into the list and try them until you find one you like. Here's the list:

for J in `elinks http://bunger.us/rez.xml`; do \
  X=`echo $J | cut -dx -f1`; Y=`echo $J | cut -dx -f2`;\
  for K in 60 75 80 120; do \
    gtf $X $Y $K | grep Modeline; \
done; done

And here's the sledgehammer that populates the list:

for J in `elinks http://bunger.us/rez.xml`; do \
  X=`echo $J | cut -dx -f1`; Y=`echo $J | cut -dx -f2`;\
  for K in 60 75 80 120; do \
    L=`gtf $X $Y $K | grep Modeline | cut -d\  -f4-99`;\
    M=`echo $L | cut -d\  -f1`; \
    xrandr --newmode $L; xrandr --addmode VGA-0 $M;\
done; done

Worst Day Ever!

Was sick all weekend, didn't leave the house. Had to go to a class in downtown DC this morning. This is third in a series that has already gone pretty bad. Got up an hour early, but ended up leaving the house fifteen minutes late. Started the car and:

Just more than an 1/8 tank of gas

Yeah... I can make it... Twenty miles to the Metro station. I'll get gas on the way home.

Got to the Metro station okay.

But I'd forgotten my Smartrip automated train ticket!

Can't very well drive thirty minutes home and thirty minute back to get it, I'll have to buy another. All the Smartrip kiosks are down. Only paper tickets available.

No big deal. I can buy a paper ticket. One problem... You can't pay for parking with a paper ticket, only with the automated Smartrip debit card. No cash... Only the single purpose debit card. And, no, I don't know why you can't pay for parking with a paper ticket.

Okay. I buy a day pass for $9.00 and will get a Smartrip card when I get to downtown. Turnstile won't accept my paper ticket. Everybody else is getting through? I walk toward one of the attendants, but don't even get a chance to open my mouth. She says:

9:30

Oh yeah, that's right... Day passes are only good 9:30AM to midnight. So I buy a $10 round trip ticket. And get on the train. And ride to downtown. And get off the train. And look for a kiosk to buy the Smartrip card. And it's down.

No time to worry, got to walk to class. What was the address? It's in my smart phone, in my case...

Crap, left my phone at home.

I've been to the building before, just look for landmarks. There it is. Made it on time.

But the vendor doesn't have custom curriculum for this particular class. I did bring my copy of the manual from the British Office of Government and Commerce, but the material they are using is from the Netherlands. It was translated from the Queen's English, to Dutch, then back to English. And the instructor is Indian. So much for all that time I spent learning German. And Italian.

So at lunch, I use the day pass to ride to Metro Center and buy a Smartrip preloaded with $5 to pay my parking fee of $4.75. And I didn't run out of gas.

So I guess the day wasn't that bad.

And sorry for the plagiarism, Jeff.

Remove XML Comments with sed

This should work for HTML also:

sed '/<!--/,/-->/d' /target/file.xml

Use sed -i save the changes back into the file.