Removing Old Linux Kernels

Today, I had trouble removing an obsolete kernel from my workstation. It should have been simple enough, but I tried to use yum erase rather than rpm -e, and kept running into errors. That is obviously the bad news, so let's make sure to report the good news: YUM is such an improvement over RPM alone, that it is smart enough to know which kernels are obsolete. For instance:

# rpm -qa kernel
kernel-2.6.32-279.el6.x86_64
kernel-2.6.32-279.19.1.el6.x86_64
kernel-2.6.32-279.22.1.el6.x86_64
# uname -r
2.6.32-279.22.1.el6.x86_64
# yum erase kernel
<snip>
Removing:
kernel   x86_64   2.6.32-279.el6
kernel   x86_64   2.6.32-279.19.1.el6
Is this ok [y/N]:

First, we determine the machine has three kernels. Second, we see that that it is running the most recent version, dot-22. Finally, YUM demonstrates that it is smart enough to erase the two old kernels, but not the current kernel.

One small problem: I don't want to remove dot-19 because I have a driver problem with dot-22. I only want to remove dot-null. Here's the trick:

# yum list kernel
Loaded plugins: refresh-packagekit, security
Installed Packages
kernel.x86_64   2.6.32-279.el6
kernel.x86_64   2.6.32-279.19.1.el6
kernel.x86_64   2.6.32-279.22.1.el6
# yum erase kernel-2.6.32-279.el6

The critical success factors are to drop the arch and t0 add a dash(-) between the package name and the version number.

Good-bye Gnome

In the latest machine re-org, I reloaded my workstation with Fedora 15. I don't like it. I think they've made several wrong turns, the single biggest being their implementation of Gnome 3.

With Gnome 3, they have set a minimum acceptable video hardware platform. That's fine, because they provided a fallback mode in case the machine does not support 3D rendering. Accept for one minor problem... most of the desktop's features do not work in fallback mode.

For instance... The ability to exit the GUI. Yes: there is no logout, quit, exit, stop, leave, or otherwise feature. The only way out is to open a terminal and init 3. If they missed something as important as an exit function, just image what else they missed. (Hint: allot!)

So, I'm trying to switch to KDE and it looks promising. As a matter of fact, I have found a fix for my single biggest complaint with KDE. I am so use to highlighting text in a terminal window and pressing "Shift-Insert", that for five years, I have refused to use KDE because it required the extra step of "Ctrl-Insert".

On the taskbar, near the clock is a scissors icon. Left click, select Configure Klipper, and click "Synchronize contents of the clipboard and the selection". Now it works the way I want!

Linux Rescue for VM on XenServer

I finally figured out how to rescue a Linux VM running on a Citrix XenServer host, an believe it or not, it is completely unintuitive! First, power off the VM, though if you're going to rescue mode... you're probably "down" already. Second, mount the rescue media (CD/DVD). Third, make sure the VM is highlighted on the left pane of XenCenter.

Here's the trick: Across the top of the XenCenter menu, select VM, and Start/Shut down, and Start in Recovery Mode. The machine should boot from the rescue media. Proceed per rescue SOP.

Be forewarned... For some reason, booting is incredibly slow.

SSH Tunneliung of X11 Apps

I had occasion to finally test something I've been wondering for a while: What is the minimal configuration to allow and X11 application to tunnel through SSH? First, this procedure assumes you have a workstation that can display X11 applications. This can either be a Linux desktop or a Windows machine running Xming or another lesser X client.

Second, install a system with only the core or base packages, possibly by building the system through kickstart. Third, edit the /etc/ssh/sshd_config and make sure X11Forwarding is set to yes. Reload if needed.

Next install a simple graphical application; for my test I used xclock:

yum install -y xorg-x11-apps

Attempt to run:

ssh 192.168.1.1 -X xclock
Error: Can't open display

In this case a failure is what we expect.

Conventional wisdom says we need to install the entire "X11 Window System" group, which will grab almost 100 packages. Instead install one RPM:

yum install -y xorg-x11-xauth
ssh 192.168.1.1 -X xclock
Warning: <snip> (repeated several times)

...but behold! A glorious xclock. The errors are from not loading fonts on the remote machine. Oddly, if we install xterm, it also complains, yet it works just fine.

A side note on this procedure: To prevent from issuing the -X (or -Y) with the SSH command line, change /etc/ssh/ssh_config, adding:

ForwardX11 yes

Adding gedit will require 57 more packages.
Adding kedit will require 68 more packages.
Best choice: gvim, requiring three packages.

yum install -y xorg-x11-fonts-Type1 xorg-x11-fonts-misc
yum install -y gvim

Mysterious Linux Permission Dots

Sometime mid last year, something strange happened: a dot appeared in the permission string of the Fedora distributions. I ask around, and no one knew where they came from, or what they were. Now I know, but first, let me show you what I'm talking about:

$ touch test.txt
$ ls -l test.txt
-rw-rw-r--. 1 doug doug 0 Feb 27 20:21 test.txt

It's hard to see, but its the eleventh character in the permission string:

1        Type of object, ex: d for dir or l for link
2-4     Permissions for owner
5-7     Permissions for group
8-10   Permissions for others
11      Mystery dot

Originally, I though this was an ext4 thing, but when I mount an ext3 under Fedora 13, I still get the dot.

Turns out, the mkfs commands have been modified to set ACLs on by default, and the dot is a place holder to represent an empty ACL. Previously, an empty ACL was not represented, as all ACLs are empty by default. Try this:

$ sudo tune2fs -l /dev/sda6 | grep options
Default mount options: user_xattr acl
$ setfacl -m u:apache:r test.txt
$ ls -l test.txt
-rw-rw-r--+ 1 doug doug 0 Feb 27 20:21 test.txt

By executing a setfacl command (as in set file ACL... ACL is pronounced like "ack ull") we change the dot to a plus, which tells us the ACL is no longer empty.

$ getfacl test.txt
# file: test.txt
# owner: doug
# group: doug
user::rw-
user:apache:r--
group::rw-
mask::rw-
other::r--

If we blank the ACL, the plus is gone, and the dot is back:

$ setfacl -b test.txt
$ ls -l test.txt
-rw-rw-r--. 1 doug doug 0 Feb 27 20:21 test.txt

Authenticated ESMTP over SSL with Sendmail

I've been wanting to get this running, but kept running into the same problem. Imagine my chagrin to find the problem was something simple. At least it's simple once you understand how it's suppose to work.

First the basics, we open a port on the firewall, and configure Sendmail. Check out Denie's Blog for an explanation of the steps. Assuming you've got a running Sendmail / Dovecot server, here's the quick and dirty:

yum install -y cyrus-sasl
chkconfig saslauthd on
service saslauthd start
echo "mydomain.xyz RELAY" >> /etc/mail/access

All authentication must always be encrypted. Setup SSL by piggybacking on Dovecot:

mkdir -p /etc/pki/sendmail/{certs,private}
cd /etc/pki/sendmail
ln -s ../../dovecot/certs/dovecot.pem certs/sendmail.pem
ln -s ../../dovecot/private/dovecot.pem private/sendmail.pem

And changes to /etc/mail/sendmail.mc

dnl Authenticated send from mobile dnl
DAEMON_OPTIONS(`Port=1234, Name=MTA, M=Ea')dnl
dnl No anonymous logins (y) dnl
define(`confAUTH_OPTIONS', `A y')dnl
TRUST_AUTH_MECH(`LOGIN')dnl
define(`confAUTH_MECHANISMS', `LOGIN')dnl
define(`confCACERT_PATH',`/etc/pki/tls/certs/')dnl
define(`confCACERT',`/etc/pki/tls/certs/ca-bundle.crt')dnl
define(`confSERVER_CERT',`/etc/pki/sendmail/certs/sendmail.pem')dnl
define(`confSERVER_KEY',`/etc/pki/sendmail/private/sendmail.pem')dnl

Last step, restart Sendmail, test, and it was the test that I was messing up on. It always failed. To test, use telnet:

$ telnet 1.2.3.4 1234
Trying 1.2.3.4...
Connected
Escape character is '^]'.
220 ESMTP Sendmail 8.14.4/8.14.4; Thu, 24 Feb 2011 12:57:44 -0500
ehlo localhost
250-Hello pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-AUTH LOGIN
250-STARTTLS
250-DELIVERBY
250 HELP

First, we're looking for AUTH LOGIN. We need to send our username and password... but we have to send it using BASE64 encoding. This is security through obscurity, in the respect that anyone can crack BASE64. To change our clear text to the expected format, we use something like Ostermiller's Javascript utility. The next trick is that your username is the sending email address which has to be in the format of:
      unixname @ mydomain.xyz
...where the domain is the one we added for RELAY to /etc/mail/access. In the open telnet session:

AUTH LOGIN
334 VXNlcm5hbWU6
ZG9zz0BzzW5zzXIzzXM=
334 UGFzc3dvcmQ6
bWzzbSzzNQ==
235 2.0.0 OK Authenticated

We're in. Make sure to add the login name in the client in the same A@B.C formal.

We need to make a last change to force the rejection of clear text passwords. Back to the /etc/mail/sendmail.mc

dnl Only authenticate across SSL (p) dnl
define(`confAUTH_OPTIONS', `A p y')dnl

Add the "p" option, and restart Sendmail.

Restrict Concurrent Remote Logins

I stumbled upon an interesting puzzle. I was asked to configure a system that would only allow a user to SSH from one remote address at a time, but allow multiple logins from that location. Furthermore, they can login from where ever they want, but never from two locations. Oh, and the restriction can't block the TTYs or Xterms.

I'd never heard of a scenario like that. There are lots of bells and whistles to lock down a system, but this one caught me off guard. After a few minutes searching the Interweb, I decided to whip out a hack:

#!/bin/sh
#
# Restrict concurrent logins from multiple locations
#

MYT=`tty | sed "s~/dev/~~"`
MYL=`who | grep "$MYT" | awk '{print $NF}' | sed 's/[()]//g'`
MYC=`who | grep "$USER.*\..*\." | grep -vc "$MYL"`
if [ "$MYC" -gt 0 ]; then
  echo "Logged in on $MYT from $MYL"
  echo "Other remote locations:"
  who | grep "$USER.*\..*\." | grep -v $MYL | \
    awk '{print $NF}' | sed 's/[()]//g' | sort -u | xargs echo " "
  echo "Too many remote logins. Good bye."
  logger -p authpriv.warn "Killed remote login: $MYT $MYL"
  ps | grep -m 1 "$MYT" | awk '{print $1}' | xargs kill -9
  #fuser -k `tty`
fi

Here's the flow:
  1. Determine our current TTY
  2. Get our remote address (client address)
  3. Are we logged in from another address...
      where the other address has two dots...
      and is not our own address
  4. If so, print all sorts of helpful information
      Note: comment "echo" lines in the wild
  5. Log the event
  6. Kick the bastard off
    fuser *should* have worked :(
Save it as /etc/profile.d/location.sh and it will automatically be called after SSH authentication.

*** Update ***
A big mistake to avoid... Don't use the exit command in any script in the /etc/profile.d directory. This will cause the login process to exit, not the script.

A Second Apache Instance With YUM

I ran into a situation where I needed two separate instances of the Apache HTTPD service on the same server. I couldn't simply virtual host the second site, because it needed a radically different configuration from the first instance. My first reaction was that I would use YUM to install the first instance, and then snag the Apache source and compile the second instance. The problem with this was that the two instances would be different versions: the first easily patched and upgraded with YUM and the second being an administrative nightmare of patching and recompiling.

To prevent the overhead of administering the second instance, I started investigating an old RPM option I'd never used: --relocate. It turns out this option was the opposite of what I had expected, in that it moved the first instance, rather than install a second. And besides, using RPM manually was only incrementally better than the original idea.

So what about YUM? There is an option for --installroot=/path. Seems like what I wanted: instead of distributing files based on system root to /etc, /usr, /var, lets put the httpd files in an tree under /opt. What happened when I ran the command surprised me:

yum install httpd --installroot=/opt
<snip>
Transaction Summary
=============================
Install 77 Package(s)
Update 0 Package(s)
Remove 0 Package(s)

Total download size: 64 M
Is this ok [y/N]:

This thing is not going to only install the httpd binaries, libraries, and config files... but every dependency... which already exists on the system! And its going to require 64M of disk space!

Oh, wait... I've got like 250G of free space. Do I really care if it take 64M? No! And so, I answered "Y". What did we end up with?

ls /opt
bin dev home lib64 mnt proc sbin srv tmp var
boot etc lib media opt root selinux sys usr

Ouch! That's ugly. Looks like the better (cleaner, prettier) choice would have been:

yum install httpd --installroot=/opt/httpd-2i

For the sake of simplicity, through the miracle of virtualization, lets just consider that fixed.

Lets see what we got:

/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)

What about an update? I added a repo file to include the updates directory on the satellite server.

yum update -y httpd
<snip>
/usr/sbin/httpd -v
Server version: Apache/2.2.9 (Unix)
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)

As expected for the base install, but no love from the second instance.

yum update -y httpd --installroot=/opt/httpd2i/
Setting up Update Process
No Packages marked for Update

Still no good. As a matter of fact, nothing seemed to work. So, as a workaround, I tried this:

yum install httpd -y --installroot=/opt/httpd2i-2/
rsync -Pr httpd2i-2/* httpd2i/ --update
/opt/httpd2i/usr/sbin/httpd -v
Server version: Apache/2.2.9 (Unix)
rm -rf /opt/httpd2i-2

In a nutshell, create a third instance, and copy the third instance over the second, hoping not to overwrite any configuration files in the process.

Does this solve the original problem? Sort of. Is it easier than recompiles? Its faster. Just one more problem... As is, the new Apache does not run. Looks like we need some more hacking. Stay tuned for part 2.

*** Update ***
On second though... I'll just recompile. It turns out there are some references to that path in the RedHat binaries. That's bad form on their part, and they should be ashamed, but by the time I figure out how to hack this, the recompile will be done.

So, no part two. Just snag the binaries and be done with it. That doesn't mean that this feature is useless. It just means that it didn't solve this problem.

Fedora and Screen Resolution: Pt2

Turns out, the xrandr info is not persistent across reboots. The obvious solution was to add the needed lines to the /etc/rc.local, but xrandr will only execute if the X engine is online. The solution was to create a script and drop into the X window systems initialization sequence:

cat /etc/X11/xinit/xinitrc.d/setres.sh
#!/bin/sh
xrandr --newmode "1280x1024" 108.88 1280 1360 1496 1712 1024 1025 1028 1060 -HSync +Vsync
xrandr --addmode VGA-0 "1280x1024"
xrandr -s "1280x1024"

Notice the scripts path.

Fedora and Screen Resolution

For the last few revisions of Fedora, I've had problems with getting my physical systems to behave at the desired screen resolution. The problem is the way the video drivers attempt to detect them monitor-- it is assumed that the display hardware is an LCD. Unfortunately, some of my physical systems still use CRT's. Why? Because they work.

I don't know exactly when the change occurred, but starting with F13, my system always booted to 1024x768. Back in the day, we'd use system-config-display to set the proper resolution, but it is no longer a default component, and has become unreliable. Instead, we should use the less intuitive xrandr. So, here's what it takes to override the default resolution.

First, determine what xrandr sees. From within an X desktop environment, open a terminal window.

xrandr -q
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2944 x 1024
VGA-0 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0*
  800x600       60.3
  640x480       59.9

It knows that Screen 0 (the monitor) is capable of 2944x1024. For some reason, VGA-0 (the video card) is defaulting to 1024x768 as indicated by the asterisk.

Second, we need to configure more video modes. This requires that we feed xrandr a huge amount of information we don't have. Luckily, gtf knows what we need. Let's try to bump the resolution up one notch to 1152x864 with 60Hz refresh rate.

# gtf 1152 864 60
# 1152x864 @ 60.00 Hz (GTF) hsync: 53.70 kHz; pclk: 81.62 MHz
Modeline "1152x864_60.00" 81.62 1152 1216 1336 1520 864 865 868 895 -HSync +Vsync

We need the second line to feed to xrandr:

xrandr --newmode "1152x864_60.00" \
  81.62 1152 1216 1336 1520 864 865 868 895 -HSync +Vsync
xrandr -q
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2944 x 1024
VGA-0 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0*
  800x600       60.3
  640x480       59.9
  1152x864_60.00 (0x7e)   81.6MHz
    h: width 1152 start 1216 end 1336 total 1520 skew 0 clock 53.7KHz
    v: height 864 start 865 end 868 total 895 clock 60.0Hz

Our new mode has been staged, now it has to be connected to a video card, and activated:

xrandr --addmode VGA-0 "1152x864_60.00"
xrandr -q
Screen 0: minimum 320 x 200, current 1024 x 768, maximum 2944 x 1024
VGA-0 connected 1024x768+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0*
  800x600       60.3
  640x480       59.9
  1152x864_60.00   60.0
xrandr -s "1152x864_60.00"
xrandr -q
Screen 0: minimum 320 x 200, current 1152 x 864, maximum 2944 x 1024
VGA-0 connected 1152x864+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
  1024x768     60.0
  800x600       60.3
  640x480       59.9
  1152x864_60.00   60.0*

If you're lucky, you can still see your video output!

Oddly, it would seem the part in quotes is completely arbitrary, but GNOME's desktop resolution utility needs that field to be in the 1234x567 format. The real headache is trying to determine the available resolutions and scan rates. If you really feel daring you can pump all the possibilities into the list and try them until you find one you like. Here's the list:

for J in `elinks http://bunger.us/rez.xml`; do \
  X=`echo $J | cut -dx -f1`; Y=`echo $J | cut -dx -f2`;\
  for K in 60 75 80 120; do \
    gtf $X $Y $K | grep Modeline; \
done; done

And here's the sledgehammer that populates the list:

for J in `elinks http://bunger.us/rez.xml`; do \
  X=`echo $J | cut -dx -f1`; Y=`echo $J | cut -dx -f2`;\
  for K in 60 75 80 120; do \
    L=`gtf $X $Y $K | grep Modeline | cut -d\  -f4-99`;\
    M=`echo $L | cut -d\  -f1`; \
    xrandr --newmode $L; xrandr --addmode VGA-0 $M;\
done; done

Elegant Log Compression

Here's an elegant little one liner that I didn't expect to work. I partition pushing 99% space used, the largest culprit being daily log files. Hey weren't mine so I couldn't delete them, but I could compress them. But what about next month?

How about this:

cd /some/path/logs
for J in `ls *log?$(date +%Y)-*(expr $(date +%m) - 1)-*`; do
  ls -lh $J; tar -czf $J.tgz $J; ls -lh $J.tgz; mv $J /dev/shm;
done

The beauty of this is the embedded execution statements.

Within the backticks, are a pair of executions, one of which nests an execution.

This particular incarnation compresses last months logs. It shows the original size and the compressed size, then moves the file to a holding directory. On a real machine, I'd probably change the middle line to:

tar -czf $J.tgz $J; rm -f $J

Pop this in a cronjob and run it at "1 2 3 * *".

every: command not found

This was an interesting puzzle. I was looking at a hardening document for Linux that identified a huge number of files that needed more restrictive file permissions. Among them were /root/.bashc, /root/.bash_profile, /root/.bash_logout, and so on. It made sense-- nobody else needs to read them, yet they were 644 instead of 600.

But the doc pointed out that the user "root" might not use the bash shell. What if he was a psychopath and used csh? In that case you'd have to look for a bunch of .c* files. I immediately realized the best thing to do was to wildcard this task:

chmod 600 /root/.*

And then I moved on.

But within a few minutes... something was wrong. In another window, as user "doug", I tried to list a directory.

-bash: ls: command not found

What? I couldn't even list my home directory. As a matter of fact, I couldn't execute any command.

This is where your heart kind of skips a beat. As root I could list anything, including /bin/ls. So as root, I tried to switch to user doug:

su: warning: cannot changeto directory /home/doug:
  Permission denied
su: /bin/bash: Permission denied

Oh crap!

Eventually it occurred to me. Consider this situation:

ls -a /root
.   ..   .bashrc   .bash_logout   .bash_profile

When I used the dot-splat wild card, it must have picked up dot-dot, which would be the root directory. From there it probably reset the permissions on /bin, /etc, and so on. I just needed to reset those perms to 755.

No luck. As a matter of fact, every directory seemed to be correct. I could not see any permission that was wrong. And then... in a stroke of unparalleled genius, I tried something else. I looked at the only set of directory permissions you can never see:

# ls -ld /
drw------- 24 root root 4096 Aug 22 22:51 /

What about:

# chmod 755 /
# ls -ld /
drwxr-xr-x 24 root root 4096 Aug 22 22:51 /

And now everything works.

Whew.

In the end, however, I do have to admit one thing: The system was significantly hardened. Hard as a brick!

Installing OpenSSH from Source

I've got a cluster of Fedora 9 machines that run Apache web servers. Since they run Apache 2.2, there is no reason to upgrade the OS. Unfortunately, I stumbled on a bug with Fedora 9's implementation of key authentication with OpenSSH. It was apparently fixed in F10, but as is one of the inherent dangers of Fedora not patched in F9. Rather than kluge around with patching the RPM, I decided to install OpenSSH from source.

When you visit the OpenSSH website you want to get the portable source. I downloaded that onto the box and extracted it to /usr/local to create the openssh-5.1p1 sub. Normally the README file explains the compile sequence but in this case I had to get the instructions from the FAQ.

The first few attempts failed until I installed zlib-devel and openssl-devel. Then it was a simple case of the standard:

./configure
make; make install

This placed the binary in /usr/local/sbin, but messed up the etc structure.

All the config files were in etc and not a sub, so I created /usr/local/etc/openssh and moved all the files into the sub. This required an update to the sshd_config, however. I had to edit he HostKey parameters to include the sub in the path.

To test, we execute:

/usr/local/sbin/sshd -Dd \
  -f /usr/local/etc/openssh/sshd_config

Connect from remote. Test the keys. Bug gone. All good.

Now to symlink everything

cd /etc/
mv ssh ssh-redhat
ln -s /usr/local/etc/openssh ssh-openssh
ln-s ssh-openssh ssh
ls -ld ssh*
cd /etc/init.d
cp sshd sshd-openssh
mv sshd sshd-redhat
ln -s sshd-openssh sshd

This gives us a SysV startup script that points to the correct config files, but the wrong binaries. We need to change all the /usr/ entries to /usr/local/:

sed -i "s~/usr/~/usr/local/~" sshd-openssh

(There's actually only two lines, and the first shouldn't count.)

Oddly, on first try, it fails. The reason is that RedHat built the SysV script to check for the path of the config, but didn't provide the path. This means it fails and uses the default. Since we moved the config... it fails. The solution, which makes everything portable is the put the config path where RedHat expects it:

echo 'OPTIONS="-f /etc/ssh/sshd_config" ' > /etc/sysconfig/sshd

Optionally, recompile with the --sysconfdir=/etc/ssh such that both binaries point to the same sub.

One downside is that the binary is running unconfined by SELinux. If you're really ambitious:

chcon -t sshd_exec_t /usr/local/sbin/sshd
chcon -u system_u /etc/init.d/sshd*
chcon -t initrc_exec_t /etc/init.d/sshd*

Restart the service to confine.

Thanks for Visiting: Script Kiddy

Everybody knows that hackers fall into two categories: script kiddies and Chinese cyber warriors raised from birth to destroy the American power grid.

Well, I've got this little VM floating around the clouds of the internet. Nothing exciting. It hosts http://dougbunger.com, which is mostly 404 pages and dead links. But... its my little cloud VM, and I love it.

So all week long, somebody has been slamming my server, trying to hack in. Why? There's nothing of value. Not quite true: chances are, if they were to compromise my server, they would probably use it as a file drop for pirated media or pr0n. (And not the good kind... of either.)

I don't think its the Chinese: they are too busy hacking Google to read their dissident's email. No, its the Script Kiddies. How do I know? They are hitting the server with thousands of PHP and SQL exploits. Unfortunately, the server has neither. So, I implemented an Apache redirect:

AliasMatch ^$ /var/www/html/index.html
RedirectMatch (.*[pP]+[hH]+[pP]+.*) \
    http://english.cpc.people.com.cn
RedirectMatch (.*[sS]+[qQ]+[lL]+.*) \
    http://english.cpc.people.com.cn

I inserted two lines that evaluate the URL and redirect anyone that ask for anything containing PHP or SQL to another website. My regex was not sufficiently righteous, and redirected blank URI's, so the first line ensures you get an index page.

And where does something like http://vypress.bunger.us/sql.php redirect? Why to the Chinese Communist Party home page, of course. Their people are trained for this kind of thing. I'm sure they will appreciate the practice.

Browser Based SSH via Webshell

Lets say you need to SSH into your server, but you're not at your regular workstation. I've always recommended people carry a USB thumb drive with a toolkit of programs, such as Putty. But what if the machine you have doesn't have a USB port. No problem, you can download Putty. But what if the machine you have is a kiosk terminal that doesn't allow you to download...

Yeah, I'll admit it sound's pretty far fetched, but I have found an ultra cool package that could provide exactly such an emergency functionality: Webshell 0.9.6 It runs as a local python service and allows login via an AJAX enabled browser.

Behind the scenes, the browser client communicates with the python service, and the python service acts as an SSH client to access the local SSH service. On the surface, this could be a problem, as the browser to python connection would normally be unencrypted. This issue can be mitigated by install OpenSSL support for python. Unfortunately, the pOpenSSL package wasn't in my Fedora repo, so I had to grab it from Pbone.

I made a couple tweaks to my install. I changed the port from the default 8022:

sed -i "s/8022/???/g" webshell.py

And since we always change the SSH port of outside servers:

sed -i "s/in +' loc/in +' -p ???? loc/" webshell.py

And added some headspace to the top of the page:

sed -i "s/margin:0;/margin:25px 0px 0px 0px;/" \
  www/webshell.css

And changed the font from 10 to 12:

sed -i "s/font:10/font:12/g" www/webshell.css

Once you change the font size, you'll need to change the default background or remove the JPG for solid black.

The documentation is a little unclear on the fact that the program, by default, only listens on 127.0.0.1, so you have to launch the script with -i 0.0.0.0 to accept outside connections. Of course, you'll have to build your own SysV start script.

A side note, there are websites that run this program as a free service to let you web into their server, then hop over to yours. You probably don't want to use those free services. Sure, its SSL from you to them, and SSH from them to your server, but what's the protocol that encrypts the link between the SSL and SSH? can you say none?

Compiling Apache Without Default Modules

I have always liked the fact that RedHat and Fedora's Apache httpd RPM is compiled as a fully modular server. Yeah, you loose a couple performance points, but you have a slim footprint which allows more sessions, and there aren't unneeded subroutines waiting to be exploited. Yet, if you download Apache source and try to compile, you get 24 components added to your binary. Bloat!

To compile a slim, modular Apache, use:

./configure --enable-mods-shared=all --with-mpm=prefork \
  --disable-deflate
make; make install
/usr/local/apache2/bin/httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

But this raises an interesting question-- what if we actually want to statically compile some, but not all, modules? Maybe we want a dedicated proxy/balancer:

./configure --enable-mods-shared=all --with-mpm=prefork \
  --disable-deflate --enable-proxy=static \
  --enable-proxy-ajp=static --enable-proxy-balancer=static
make; make install
/usr/local/apache2/bin/httpd -l
Compiled in modules:
  core.c
  mod_proxy.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  prefork.c
  http_core.c
  mod_so.c

And that's what we are looking for. We'll need to get SSL on this puppy, but it's bed time, so go to sleep.

Another Tomcat Post :: SSL (Part 2)

Oh no-- Not more Tomcat SSL! Yes. But. In the immortal words of Bullwinkle J. Moose: "This time for sure!"

This entry is a follow up to a post a few days ago quaintly titled Another Tomcat Post :: SSL. Since that post, I have made a momentous discovery regarding Tomcat encryption.

There is an annoying error message that writes to catalina.out on Tomcat restart, that it turns out is relevant to why this SSL has been such a mess.

Jun 16, 2010 8:55:14 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:

This is telling us that we are not using packages optimized for Tomcat. The message can be cleared by installing the tomcat-native RPM:

yum install -y tomcat-native

On restart, the irritating little message is gone.

One of the things that this package does is to include an updated crypto stack for Tomcat that includes x509. Glory to the mighty Gods of Olympus! Once installed, we modify the SSL stanza in the server.xml file:

maxThreads="150" scheme="https" secure="true"
SSLCertificateFile="conf/custom.crt"
SSLCertificateKeyFile="conf/custom.key"
clientAuth="false" sslProtocol="TLS" />

We've removed the keystore directives and used commands to point to our x509 cert and key. Restart Tomcat.

To test:

echo | openssl s_client -connect localhost:8443 | \
grep subj

If you are not using RPMs, but Apache's Tomcat release, look in CATALINA HOME's bin directory for a tar file.

Setting XMMS As Default Player on F13

My home workstation has a dual head Vista laptop and a Linux desktop. The desktop has better speakers, so I'm using it as my office jukebox. I've always used XMMS, but ran into a small problem getting it set as the default player on F13. What I wanted, was to be able to double click an MP3 and have it start playing. What I got, was that the song would queue, but not play. Here's what I had to do to fix it:

# grep Exec /usr/share/applications/xmms.desktop
Exec=xmms -e %F
# sudo vi /usr/share/applications/xmms.desktop

And change the -e to -p. This changes XMMS's behavior from enqueue to play. For some reason, someone decided they wanted to double click to add songs to a manually executing playlist-- Every other player (including Windows Media Player!) uses drop and drag to add songs to the playlist.

BTW: This is what was originally posted that ended up on a file by file basis rather than globally.

1. Open the Music folder (or any location that has an MP3.)
2. Right click an MP3 file and select Open with Other Application.
3. Find and highlight XMMS.
4. Expand the option to Use a custom command.
5. Add " -p " to the xmms command. (The spaces are important.)
6. Check Remember this application.
7. Click Open.

Happy Fedora 13 Day

In keeping with my standing policy, I've skipped a version, and jumped from Fedora 11 to 13. I was quite disappointed, however, that Xen Dom0 is not included.

* Disk Druid has changed, allowing for safer isolation of disks that should not be formatted. Unfortunately, I had problems getting LVM to work.
* Once again I loaded KDE, and found it beautiful, then promptly did away with it. I just can't stand Konsole-- I've got to have fast cut and paste.
* Looks like Plymouth for ATI Radeon is working, but I'm back to not being able to get the resolution beyond 1024x768.
* NIS still doesn't work out of the box, but I've got to move to Kerberos anyway.
* And NetworkManager... It just keeps getting worse and worse.
* The Grub kernel line is significantly more complicated, because it seems as if it is being ordered to NOT load modules.

I'll reload again tomorrow and we'll see if there are any new applications.

More XenServer Paravirt Adventures

While working with someone else about running Linux under Citrix Xenserver 5, I ran across an interesting paradox. Daniel points out that the ability to run Linux as a DomU has been moved into the kernel, no longer requiring the kernel-xen RPM.

Yet, when I tried to install F11, on XenServer as a paravirt VM using the "CentOS 5.3 x64" template, it failed. When I clicked the Logs tab in XenCenter, I got:

Unable to access a required file in the the specified repository:
file:///xxxyyyzzz/images/xen/vmlinuz

The template is expecting the standard install image format.

Tried a few other templates with the same result. The only way I was able to get it to run was by installing using "Other install media", which meant it was running HVM.

The paradoxical part of this, is that Fedora implies that it can be done, yet when attempted on a Fedora Xen Dom0, an F11 paravirt VM fails:

libvir: Xen error : Domain not found:
    xenUnifiedDomainLookupByUUID
Invalid URL location given: [Errno 14] HTTP Error 404

In a way, its the message XenServer threw: I can't find the specially compiled paravirt kernel.

My guess at this point is that there has to be a new installer that understands how to use the newly compile options. Looks like there is some more research to be done. Stay tuned.