Wednesday, June 23, 2010

Compiling Apache Without Default Modules

I have always liked the fact that RedHat and Fedora's Apache httpd RPM is compiled as a fully modular server. Yeah, you loose a couple performance points, but you have a slim footprint which allows more sessions, and there aren't unneeded subroutines waiting to be exploited. Yet, if you download Apache source and try to compile, you get 24 components added to your binary. Bloat!

To compile a slim, modular Apache, use:
./configure --enable-mods-shared=all --with-mpm=prefork \
  --disable-deflate
make; make install

/usr/local/apache2/bin/httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c
But this raises an interesting question-- what if we actually want to statically compile some, but not all, modules? Maybe we want a dedicated proxy/balancer:
./configure --enable-mods-shared=all --with-mpm=prefork \
  --disable-deflate --enable-proxy=static \
  --enable-proxy-ajp=static --enable-proxy-balancer=static

make; make install
/usr/local/apache2/bin/httpd -l
Compiled in modules:
  core.c
  mod_proxy.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  prefork.c
  http_core.c
  mod_so.c
And that's what we are looking for. We'll need to get SSL on this puppy, but it's bed time, so go to sleep.

Friday, June 18, 2010

Off Peak Energy Usage

I may have complained about this before, but since nobody has fixed this yet, I'll complain again. One of my favorite tech news sites was running a story about "smart houses" and commented on energy can cost 10 times more during peak hours. The appliances in the smart house "will be able to automatically delay its actions until off-peak hours."

You know what... I don't care! It doesn't save me any money.

I get charged a flat rate, nights, days, weekends. How is my washing machine choosing to wash my clothes later because it costs less helping me? It's not! Who is it helping? The global ecosystem? BRRRRAP! Wrong answer, you naive twit-- the power company's profit margin. That's it, nobody else.

Cost savings are not passed to the consumer, they pad the Wall Street coffers. If the power companies had invested in their infrastructures over the past four decades, we'd already have a smart grid. But noooooooo. They pocketed the profits and have left the consumers to deal with their short-sightedness.

So, when I'm ready to was my clothes, I'm doing it. If I overload the grid, and brown you out, too bad. You should have installed a point of use energy system... or at least a UPS. Like me.

Wednesday, June 16, 2010

Another Tomcat Post :: SSL (Part 2)

Oh no-- Not more Tomcat SSL! Yes. But. In the immortal words of Bullwinkle J. Moose: "This time for sure!"

This entry is a follow up to a post a few days ago quaintly titled Another Tomcat Post :: SSL. Since that post, I have made a momentous discovery regarding Tomcat encryption.

There is an annoying error message that writes to catalina.out on Tomcat restart, that it turns out is relevant to why this SSL has been such a mess.
Jun 16, 2010 8:55:14 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:
This is telling us that we are not using packages optimized for Tomcat. The message can be cleared by installing the tomcat-native RPM:
yum install -y tomcat-native
On restart, the irritating little message is gone.

One of the things that this package does is to include an updated crypto stack for Tomcat that includes x509. Glory to the mighty Gods of Olympus! Once installed, we modify the SSL stanza in the server.xml file:
maxThreads="150" scheme="https" secure="true"
SSLCertificateFile="conf/custom.crt"
SSLCertificateKeyFile="conf/custom.key"
clientAuth="false" sslProtocol="TLS" />
We've removed the keystore directives and used commands to point to our x509 cert and key. Restart Tomcat.

To test:
echo | openssl s_client -connect localhost:8443 | \
grep subj
If you are not using RPMs, but Apache's Tomcat release, look in CATALINA HOME's bin directory for a tar file.

Sunday, June 06, 2010

XenServer System Alerts From the Future

I got the system alert on my XenServer this weekend. Its telling me that on the 16th there was a set of updates released. Except, its the 6th. So this system alert hasn't happened yet. Or mamybe they are going to release the updates on the 16th. Nope, the updates are there.

I'm soooooo confused.

Thursday, June 03, 2010

Another Tomcat Post :: SSL

Yeah, I'm about tired of Tomcat, too. But this is new and improved Tomcat: Now with OpenSSL. And we know how much I like OpenSSL.

Okay, simple stuff, first. When you install the mod_ssl RPM, it creates a dummy cert. Lets nuke it and create our own:
cd /etc/pki/tls
mv private/localhost.key private/localhost.key.rpm
mv certs/localhost.crt certs/localhost.crt.rpm
openssl genrsa 2048 -out custom.key
openssl req -new -nodes -subj /O=doug \
  -key custom.key -out custom.csr
And the CSR get sent to the non-existant CA... So, fudge it:
openssl x509 -noout -text -signkey custom.key \
  -in custom.csr -out custom.pem
Distribute:
cd private; mv ./custom.key .
ln -s custom.key private.key; cd ..
cd certs; mv ../custom.{csr,pem} .
ln -s custom.pem custom.crt
ln -s custom.pem localhost.crt
service httpd reload
And, yes, memorize *all* that crap.

Test httpd:
echo | openssl s_client -connect localhost:443 | \
  grep subj

Getting Tomcat to work with SSL reminds me of the chorus from an "Offspring" song, Stuff Is Messed Up.

Tomcat requires our cert be converted from x509 to pkcs12. This is not difficult, but there are two critically important issues with the following command. The assigned name must be 100% unique across all files in the working directory. As such, make sure you do this next section in an empty directory.

The second issue is that you will be prompted for a password. It must be more than six characters, even though it will accept smaller, including NULL. Would it surprise you to hear that ultimately your password is going to be coded on the system in clear text? The default clear text password is "changeit".
openssl pkcs12 -export -name unique \
  -in /etc/pki/tls/certs/custom.crt \
  -inkey /etc/pki/tls/private/custom.key \
  -out custom.p12
Ready for another puzzle? Tomcat needs another component called a keystore. Beware: This command assumes your goal is to compile all the pkcs12 files in the working directory. Wait-- Don't assume I said something that I didn't: the command does not source *.p12, it evaluates all the files in the directory and if a file is a pkcs12 file, it compiles it. That's why we're in a nearly empty directory.

And remember the password we entered a moment ago? The keystore's password must match. Oh... and the Tomcat SSL documentation is wrong.
keytool -genkeypair -keystore custom.jks \
  -alias unique -dname O=doug
Notice that there is no -in and what was a pkcs12 name is now the jks alias.

*** Updated 6/16/2010 ***
I have since learned the steps above do not work as I thought. The correct next step is not genkeypair, but instead:
keytool -importkeystore -v -srcstoretype pkcs12 \
  -srckeystore custom.p12 -destkeystore custom.jks

Almost home... Tell Tomcat where to find the keystore, cert, and key by adding the following to the server.xml, just above the line that contains "sslProtocol":
maxThreads="???" scheme="https" secre="true"
keystore="conf/custom.jks" keystorePass="changeit"
clientAuth="false" sslProtocol="TLS" />
Before you save the file, make sure the stanza you just edited is not commented out by a set of <!-- --> symbols.

Symlink the original key and cert back to Tomcat's conf directory. Restart (reload) Tomcat. Test:
echo | openssl s_client -connect localhost:8443 | \
  grep subj

What a mess, but at least it works.

Wednesday, June 02, 2010

Tomcat Load Balancing via AJP Module

I've been playing with Tomcat in my spare time and have had some fun with the load balancing module, which happens to be implemented with the help of our friend Apache HTTPD. The basic premise is that we point our traffic at an HTTPD instance and he disperses the traffic to a set of worker nodes. This can be done via the standard http protocol or the Tomcat ajp protocol.

I like this config for /etc/httpd/conf.d/proxy_ajp.conf:
<Proxy balancer://cluster-http>
ProxySet lbmethod=bytraffic nofailover=on
ProxySet stickysession=JSESSIONID timeout=15
BalancerMember http://tomcat1:8080 \
    retry=120 loadfactor=1
BalancerMember http://tomcat2:8080 \
    retry=120 loadfactor=1
BalancerMember http://tomcat3:8080 \
    retry=120 loadfactor=1 lbset=1
</Proxy>
ProxyPass /sample balancer://cluster-http/sample
In the ProxySet lines we define basic values, the only one of which is interesting is timeout. This determines how long a device has to respond before it considered "down". Next we list the nodes, in this case three.

Oddly, the timeout can be specified cluster-wide, but the retry, which specifies how often we check to see if "down" nodes are up, is listed individually. (It is possible to list timeout individually.) The loadfactor determines the weighting for each node.

The fun value is lbset. This one effectively allows the specification of a hotspare. In the above example, all "0" get hit all the time, and "1" gets no traffic. If all "0" nodes go down, "1" gets traffic.

Ready for sexy? Add this:
<Location /balancer-manager>
SetHandler balancer-manager
</Location>
ProxyPass /balancer-manager/ !
Now you have an interactive, web based, management screen: