XenServer System Alerts From the Future

I got the system alert on my XenServer this weekend. Its telling me that on the 16th there was a set of updates released. Except, its the 6th. So this system alert hasn't happened yet. Or mamybe they are going to release the updates on the 16th. Nope, the updates are there.

I'm soooooo confused.

Another Tomcat Post :: SSL

Yeah, I'm about tired of Tomcat, too. But this is new and improved Tomcat: Now with OpenSSL. And we know how much I like OpenSSL.

Okay, simple stuff, first. When you install the mod_ssl RPM, it creates a dummy cert. Lets nuke it and create our own:

cd /etc/pki/tls
mv private/localhost.key private/localhost.key.rpm
mv certs/localhost.crt certs/localhost.crt.rpm
openssl genrsa 2048 -out custom.key
openssl req -new -nodes -subj /O=doug \
  -key custom.key -out custom.csr

And the CSR get sent to the non-existant CA... So, fudge it:

openssl x509 -noout -text -signkey custom.key \
  -in custom.csr -out custom.pem

Distribute:

cd private; mv ./custom.key .
ln -s custom.key private.key; cd ..
cd certs; mv ../custom.{csr,pem} .
ln -s custom.pem custom.crt
ln -s custom.pem localhost.crt
service httpd reload

And, yes, memorize *all* that crap.

Test httpd:

echo | openssl s_client -connect localhost:443 | \
  grep subj

Getting Tomcat to work with SSL reminds me of the chorus from an "Offspring" song, Stuff Is Messed Up.

Tomcat requires our cert be converted from x509 to pkcs12. This is not difficult, but there are two critically important issues with the following command. The assigned name must be 100% unique across all files in the working directory. As such, make sure you do this next section in an empty directory.

The second issue is that you will be prompted for a password. It must be more than six characters, even though it will accept smaller, including NULL. Would it surprise you to hear that ultimately your password is going to be coded on the system in clear text? The default clear text password is "changeit".

openssl pkcs12 -export -name unique \
  -in /etc/pki/tls/certs/custom.crt \
  -inkey /etc/pki/tls/private/custom.key \
  -out custom.p12

Ready for another puzzle? Tomcat needs another component called a keystore. Beware: This command assumes your goal is to compile all the pkcs12 files in the working directory. Wait-- Don't assume I said something that I didn't: the command does not source *.p12, it evaluates all the files in the directory and if a file is a pkcs12 file, it compiles it. That's why we're in a nearly empty directory.

And remember the password we entered a moment ago? The keystore's password must match. Oh... and the Tomcat SSL documentation is wrong.

keytool -genkeypair -keystore custom.jks \
  -alias unique -dname O=doug

Notice that there is no -in and what was a pkcs12 name is now the jks alias.

*** Updated 6/16/2010 ***
I have since learned the steps above do not work as I thought. The correct next step is not genkeypair, but instead:

keytool -importkeystore -v -srcstoretype pkcs12 \
  -srckeystore custom.p12 -destkeystore custom.jks

Almost home... Tell Tomcat where to find the keystore, cert, and key by adding the following to the server.xml, just above the line that contains "sslProtocol":

maxThreads="???" scheme="https" secre="true"
keystore="conf/custom.jks" keystorePass="changeit"
clientAuth="false" sslProtocol="TLS" />

Before you save the file, make sure the stanza you just edited is not commented out by a set of <!-- --> symbols.

Symlink the original key and cert back to Tomcat's conf directory. Restart (reload) Tomcat. Test:

echo | openssl s_client -connect localhost:8443 | \
  grep subj

What a mess, but at least it works.

Tomcat Load Balancing via AJP Module

I've been playing with Tomcat in my spare time and have had some fun with the load balancing module, which happens to be implemented with the help of our friend Apache HTTPD. The basic premise is that we point our traffic at an HTTPD instance and he disperses the traffic to a set of worker nodes. This can be done via the standard http protocol or the Tomcat ajp protocol.

I like this config for /etc/httpd/conf.d/proxy_ajp.conf:

<Proxy balancer://cluster-http>
ProxySet lbmethod=bytraffic nofailover=on
ProxySet stickysession=JSESSIONID timeout=15
BalancerMember http://tomcat1:8080 \
    retry=120 loadfactor=1
BalancerMember http://tomcat2:8080 \
    retry=120 loadfactor=1
BalancerMember http://tomcat3:8080 \
    retry=120 loadfactor=1 lbset=1
</Proxy>
ProxyPass /sample balancer://cluster-http/sample

In the ProxySet lines we define basic values, the only one of which is interesting is timeout. This determines how long a device has to respond before it considered "down". Next we list the nodes, in this case three.

Oddly, the timeout can be specified cluster-wide, but the retry, which specifies how often we check to see if "down" nodes are up, is listed individually. (It is possible to list timeout individually.) The loadfactor determines the weighting for each node.

The fun value is lbset. This one effectively allows the specification of a hotspare. In the above example, all "0" get hit all the time, and "1" gets no traffic. If all "0" nodes go down, "1" gets traffic.

Ready for sexy? Add this:

<Location /balancer-manager>
SetHandler balancer-manager
</Location>
ProxyPass /balancer-manager/ !

Now you have an interactive, web based, management screen:

Setting XMMS As Default Player on F13

My home workstation has a dual head Vista laptop and a Linux desktop. The desktop has better speakers, so I'm using it as my office jukebox. I've always used XMMS, but ran into a small problem getting it set as the default player on F13. What I wanted, was to be able to double click an MP3 and have it start playing. What I got, was that the song would queue, but not play. Here's what I had to do to fix it:

# grep Exec /usr/share/applications/xmms.desktop
Exec=xmms -e %F
# sudo vi /usr/share/applications/xmms.desktop

And change the -e to -p. This changes XMMS's behavior from enqueue to play. For some reason, someone decided they wanted to double click to add songs to a manually executing playlist-- Every other player (including Windows Media Player!) uses drop and drag to add songs to the playlist.

BTW: This is what was originally posted that ended up on a file by file basis rather than globally.

1. Open the Music folder (or any location that has an MP3.)
2. Right click an MP3 file and select Open with Other Application.
3. Find and highlight XMMS.
4. Expand the option to Use a custom command.
5. Add " -p " to the xmms command. (The spaces are important.)
6. Check Remember this application.
7. Click Open.

OpenSSL: Love At Last

No... Not even close. It is so counter-intuitive, needlessly complicated, and maddeningly confusing. Thus forcing me to cheat.

Determine a website's SSL cert expiration date:

echo "" | openssl s_client -connect mail.google.com:443 \
  2> /dev/null | openssl x509 -noout -text | \
  grep After

Verify a file is a key:

openssl rsa -noout -check -in localhost.xxx

Find a key file that is mislabeled:

for J in `find . -type f`; do echo $J; \
  openssl rsa -noout -text -in $J 2> /dev/null | grep Pri; \
done

Verify a file is a certificate:

openssl x509 -noout -in localhost.xxx -enddate

Find a cert file that is mislabeled:

for J in `find . -type f`; do echo $J; \
  openssl x509 -noout -enddate -in $J 2> /dev/null; \
done

Verify the key matches the cert:

[ `openssl rsa -noout -modulus -in localhost.key` \
  == `openssl x509 -noout -modulus -in localhost.crt` \
] && echo yes || echo no

(Remember that those are back-tics.)

View a PKCS12 binary file:

openssl pkcs12 -info -nodes -in localhost.p12

Glorious Peoples Shuttle of Greatness in Space

Behold, comrades: the Soviet space shuttle Buran. After its single mission, I'd heard this pentacle of communist engineering had been retired, but look what I found (with some help) on Google maps. No street view... too sad.



View Larger Map

Happy Fedora 13 Day

In keeping with my standing policy, I've skipped a version, and jumped from Fedora 11 to 13. I was quite disappointed, however, that Xen Dom0 is not included.

* Disk Druid has changed, allowing for safer isolation of disks that should not be formatted. Unfortunately, I had problems getting LVM to work.
* Once again I loaded KDE, and found it beautiful, then promptly did away with it. I just can't stand Konsole-- I've got to have fast cut and paste.
* Looks like Plymouth for ATI Radeon is working, but I'm back to not being able to get the resolution beyond 1024x768.
* NIS still doesn't work out of the box, but I've got to move to Kerberos anyway.
* And NetworkManager... It just keeps getting worse and worse.
* The Grub kernel line is significantly more complicated, because it seems as if it is being ordered to NOT load modules.

I'll reload again tomorrow and we'll see if there are any new applications.

RedHat Tomcat 6 with Web Manager

The Red Hat Tomcat 6 RPM is not behaving politely. It would seem the manager should be available after install, but I'm getting a blank page. Turns out that I've had life pretty easy thus far (news to me) and someone has already done the hard work. Here's what it took for me to get Tomcat 6 working via RPM.

Starting with my "standard load" which does not include Apache:

# yum install tomcat6 tomcat6-admin-webapps

This will snag a quantity of dependencies, but will install with the web manager broken. Before starting Tomcat we will need to "fix" the web manager. While were at it, lets do some reorganizing:

# ls -l /usr/share/tomcat6/ | awk '{print $8" "$9" "$10}
bin
conf -> /etc/tomcat6
lib -> /usr/share/java/tomcat6
logs -> /var/log/tomcat6
temp -> /var/cache/tomcat6/temp
webapps -> /var/lib/tomcat6/webapps
work -> /var/cache/tomcat6/work

Okay... They tried to organize things, but I've never seen anybody put in /usr/share on a production system. Let's go with /opt:

# mkdir /opt; cd /opt; ln -s /usr/share/tomcat6 tomcat
# ln -s tomcat $(cd /usr/share/doc; ls -d tomcat6-*)
# ls -l | awk '{print $8" "$9" "$10}'
tomcat -> /usr/share/tomcat6
tomcat6-6.0.18 -> tomcat

Time to fix the manager. Web manager will ask for the user the authenticate, even though not user is allowed, by default.

# cd conf; grep manager tomcat-users.xml

One of the lines should show the user "tomcat" with the role of "manager". Notice the line is commented. Obviously we un-comment the line to allow a manager. We should now be ready:

# service tomcat6 restart

Hit the manager at something like:

http://tomcat.example.com:8080/manager/html

Fedora 10+ Kernel Modesetting (KMS)

Boring backstory: I tend to buy computers in sets, so currently, my three primary R&D machines are HPs: a set of twins, and a more powerful third. That one is now loaded with ESX, so my main Linux desktop is one of the twins. Last week, its hard drive died and I just got the replacement. Now to the real story...

I'd been running Fedora 6 or 8 to work with Xen. That project has been finished for several months, so when I installed the new drive, I loaded 11. I found, however, that I could not get the Gnome desktop to run at better than 1024x768. I had run 11 before without problems using the same monitor at 1600x1200-- but that was on "third", who is now running ESX.

I checked the twin out, and the card should have been able to run 1280x1024. I could get system-config-display to specify 1280, but the desktop would always drop to 1024. After investigating the problem, I found the culprit was KMS, or kernel modesetting. (Yes, its one word.)

The idea is that the kernel, who owns all the hardware anyway, will decide the best resolution, and the software will do as it is told. Unfortunately, it works with Intel, plays nice with nVidia, but there are a few issues with ATI. Turns our, third was nVidia, and the twins are ATI.

A feature that is closely tied to KMS is the new boot progress screen called "Plymouth". Without KMS, Plymouth is just a three color progress bar. With KMS, its a blue sun projecting solar flares. For these ATI Radeon machines, no Plymouth. This is because KMS isn't reverse compatible. As a result, Gnome looked to the kernel for the correct resolution, kernel said "don't know", and so the desktop could not be made to exceed 1024.

In the end, the solution ended up being to add a Grub argument:

nomodeset

Still no Plymouth, but when Gnome asks the kernel for the correct resolution, the response is "decide yourself". Worked for me. Other possibilities, any one of the following:

vga=795
radeon modeset=0
radeon modeset=1

Good reading:
Plymouth Graphical Boot
How To Enable Graphical Boot with Plymouth

Sudo Read Only All

I had a friend with an interesting problem: They had replicate a set of configuration files on one Linux machine to another, but she didn't have root on the old box. Thus, she couldn't read files like the /etc/securettty file, which was permission 600.

Here's where life gets strange... The customer didn't mind her looking at the box, they just didn't want her changing anything. The best way to make sure she doesn't change anything is to not give her sudo.

Rock --> You <-- Hard place.

Solution: /usr/bin/less is a read only command so lets just sudo it! Unacceptable, as there is a thirty year old hack that lets you bang out of less to a command line, sayeth information security. Easy enough to fix...

echo "username ALL=NOEXEC: NOPASSWD: /usr/bin/less" >> /etc/sudoers

The NOEXEC: prevents the "bang hack" and allows full system visibility.