Monday, December 27, 2010

An EC2 Conundrum

Whenever I would lecture on Amazon's EC2, I would point out that Amazon's internal infrastructure is (effectively) EC1, and when they need capacity, it comes from EC2. In the past few weeks, I've seen this first hand. Of course, we need to remember that this is Amazon's peak period, so a resource crunch should be expected as part of the normal patterns of business activity (BPA), but I was caught by surprise in this one respect.

The Amazon cloud, known as Amazon Web Services (AWS), is billed on three meters:
* VM Resources, such as CPU and memory
* Storage, either block volumes or web based files
* Bandwidth, both into and out of the cloud

I have a set of VMs that I launch as needed, so I am not always billed for cycles or bandwidth. When I need the VM, I don't want to have to upload all the supporting applications and data, or go through a complex configuration procedure. The solution was to grab an Elastic Block Store (EBS), which looks to a Linux VM as a disk device. I provision the VM, connect the volume, log in, mount the device, where I have a set of scripts that rebuild the application server in less than 1 minute.

Here's where I got burned: The EBS is actually a LUN on a SAN, which resides in a data center, somewhere in the world. Amazon has four regions: Virginia, California, Ireland, and Singapore. I picked Virginia. But in Virginia, they have four data centers, called availability zones, labeled A, B, C, and D. My volume is in Virgina "B". Unfortunately, they have insufficient capacity in VA-B to launch a VM, as of about 21 December.

This means I've got stuff on a disk, somewhere across the Potomac, that I can't get to, because I don't have a machine to access it. I could launch a VM in VA-C or VA-D, but there is no native mechanism to allow VMs to mount disks that live in another data center. Thus the conundrum: How do we protect against this situation?

The answer is obvious: clustered replication. Two EBS volumes in different data centers, with one VM acting as the master node, and another VM acting as the replication node. Unfortunately, this doubles the cost of the system... From $15 a month to $30 a month. Not really that much... and that assumes my data is critically important, which it isn't.

But you'd think Amazon would have provided a way to prevent this from happening. After all, its not like me paying twice as much on a monthly basis is something they'd actually want to happen.

Saturday, December 25, 2010

Schug Pinit Noir

This is a California (Sonoma Coast) wine, that I ended up with twice during the holidays. The first time was at a company gathering, the second was when I got it as a gift. Its a good wine with a prominent smokey flavor, but not high tannins.

Th company gathering was at a high end restaurant, but it was served a dining room temperature. Once I chilled the bottle I got at home, I was much more satisfied. This is also a wine that truly needs to breath to mellow. Though not expensive, it is at the close to my high end for everyday wine. Out of 10, I'll go a strong 6, pushing 7, with the right preparation.

Sunday, December 19, 2010

A Second Apache Instance With YUM

I ran into a situation where I needed two separate instances of the Apache HTTPD service on the same server. I couldn't simply virtual host the second site, because it needed a radically different configuration from the first instance. My first reaction was that I would use YUM to install the first instance, and then snag the Apache source and compile the second instance. The problem with this was that the two instances would be different versions: the first easily patched and upgraded with YUM and the second being an administrative nightmare of patching and recompiling.

To prevent the overhead of administering the second instance, I started investigating an old RPM option I'd never used: --relocate. It turns out this option was the opposite of what I had expected, in that it moved the first instance, rather than install a second. And besides, using RPM manually was only incrementally better than the original idea.

So what about YUM? There is an option for --installroot=/path. Seems like what I wanted: instead of distributing files based on system root to /etc, /usr, /var, lets put the httpd files in an tree under /opt. What happened when I ran the command surprised me:
yum install httpd --installroot=/opt
Transaction Summary
Install 77 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 64 M
Is this ok [y/N]:
This thing is not going to only install the httpd binaries, libraries, and config files... but every dependency... which already exists on the system! And its going to require 64M of disk space!

Oh, wait... I've got like 250G of free space. Do I really care if it take 64M? No! And so, I answered "Y". What did we end up with?
ls /opt
bin dev home lib64 mnt proc sbin srv tmp var
boot etc lib media opt root selinux sys usr
Ouch! That's ugly. Looks like the better (cleaner, prettier) choice would have been:
yum install httpd --installroot=/opt/httpd-2i
For the sake of simplicity, through the miracle of virtualization, lets just consider that fixed.

Lets see what we got:
/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)
What about an update? I added a repo file to include the updates directory on the satellite server.
yum update -y httpd
/usr/sbin/httpd -v
Server version: Apache/2.2.9 (Unix)
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)
As expected for the base install, but no love from the second instance.
yum update -y httpd --installroot=/opt/httpd2i/
Setting up Update Process
No Packages marked for Update
Still no good. As a matter of fact, nothing seemed to work. So, as a workaround, I tried this:
yum install httpd -y --installroot=/opt/httpd2i-2/
rsync -Pr httpd2i-2/* httpd2i/ --update
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.9 (Unix)
rm -rf /opt/httpd2i-2
In a nutshell, create a third instance, and copy the third instance over the second, hoping not to overwrite any configuration files in the process.

Does this solve the original problem? Sort of. Is it easier than recompiles? Its faster. Just one more problem... As is, the new Apache does not run. Looks like we need some more hacking. Stay tuned for part 2.

*** Update ***
On second though... I'll just recompile. It turns out there are some references to that path in the RedHat binaries. That's bad form on their part, and they should be ashamed, but by the time I figure out how to hack this, the recompile will be done.

So, no part two. Just snag the binaries and be done with it. That doesn't mean that this feature is useless. It just means that it didn't solve this problem.

Wednesday, December 15, 2010

Evil Free Cell Game: #29868

I finally beat it. Now I can go to bed.

Thursday, December 09, 2010

So, So, Sad: And its ITIL's Fault

I signed up for a series of ITIL classes with the goal of earning the ITIL Expert Certification. (They're choice of words, not mine; I'm leery of "experts", personally.) There are five classes, each with a test. Once you pass all five classes and tests, there is a sixth class and test.

Yesterday, I found out that I failed the fifth test. I was crushed! Not because I failed: I fail all the time. Constantly. As a matter of fact, I failed tests one and two, but those didn't bother me. Let me explain.

I call the testing format Three Little Kittens.

You get a case study.
Three little kittens, have lost their mittens.
You must select "the best" solution, based upon four chioces:
1. They bought gloves
2. They found their mittens
3. And they shall have no pie
4. Kittens don't wear mittens
The operative factor in this process is the fact that we have to pick "the best" solution. The answers are weighted with scores of 5, 3, 1, and 0 points. In the case study above, the answers logically break out as follows.
1. Throw money at the problem
2. A definitive solution
3. Punishment does not solve the problem
4. True, but irrelevant
Thus, the 5 point answer is "2", the 3 point answer is "1", the 1 point answer is "3", and number "4" is worth nothing, even though it is completely accurate.

I failed the first two tests because I did not personally recognize the level of dedication that is needed for the certification track. Furthermore, the class vendor, Global Knowledge, has not done a good job of setting expectations. Embarking on this process requires either significant management and project experience, or the purchase of supplemental material and several weeks of study before the class.

This certification also requires complete support from your employer. They have got to be willing to give you the time and resources to succeed. They have got to recognize the value they will receive from this process.

I have scheduled to retake the class and tests for 1 and 2. After passing tests 3 and 4, I was very confident that I understood the testing method, and the amount of preparation needed before hand. My results for test 5?
50% of answers were 5 pointers
12% of answers were 3 pointers
 0% of answers were 1 pointers
38% of answers were 0 pointers

So, whose fault is this? Doesn't matter (see justification 3 above.) But I am sad.

Monday, December 06, 2010

Why Am I Changing Light Bulbs?

Being an ecologically conscious kinda guy, about three years ago, I went through my house and replaced my incandescent bulbs with CF bulbs. It was hugely expensive because I bought good GE brand bulbs. I saw it as an investment. Not only would the bulbs save energy, but they would last a thousand years.

Yes, a thousand years, damn it. They said the bulbs lasted five times longer than "normal" bulbs. They said that even though they cost more, they don't really cost more when you consider the cost of all the bulbs you won't have to buy in the future. And you're saving energy.

Yeah, because they put out half the light of normal bulbs. One bathroom had a two bulb fixture. If you walked into the bathroom in the middle of the night, it took five minutes for the lights to power-up. (I was already "done" by then.) So, I changed one of the CF bulbs for a normal bulb. For short jobs, the incandescent bulb fires up giving us 50% light, and for jobs taking more than 15 minutes, the CF gets us up to 90%. And still saves energy.

But the CF bulb has burned out. Not the normal bulb. The CF that was suppose to last five times longer! What's the deal? It's almost like they lied to me... or something.

Oh now: lets not loose sight of what's important. I'm saving the planet. I'm being environmentally aware. The CF bulbs reduce my carbon footprint. And they contain deadly levels of mercury that is sufficiently toxic that improper disposal is criminal in some jurisdictions.

Now if you'll excuse me, I've got some endangered tigers that need to be shot.

Saturday, December 04, 2010

Oops, I've Seen All of Netflix

Yep. All of it.

Okay, not every movie in the entire Netflix inventory, but every movie that I've ever wanted to see. I'm down to watching foreign flix with subtitles. I'm to the point that I'm saying to myself: "Hey, I don't think that was all that bad. Oh sure, I purposely avoided it when it came out at the theaters since it wan't worth spending money on, and I didn't watch it on HBO or TV since I had better stuff to do, but I'll add it to my Netflix queue."

Why? I because Netflix is a flat rate service. Its a buffet. All you can eat. There was an old Huey Lewis song:
"The sign on the door said all you can eat for $1.99, but one dollars worth was all i that could stand"
(Like there is anything but old Huey Lewis songs.

So here's what I have left:
- The Good, the Bad, the Weird: A subtitled Chinese Western
- How to Train Your Dragon: What passes for a cartoon these days
- Daybreakers: Because the world needed another vampire movie
- Scott Pilgram vs The World: For the 80's video game references

Sometime in the future, they'll have:
- Despicable Me: I dream of world domination
- Inception: Speaking of dreams
- Skyline: Gotta keep an eye on the aliens

I hope somebody invents something Earth shatteringly entertaining... quick.