Authenticated ESMTP over SSL with Sendmail

I've been wanting to get this running, but kept running into the same problem. Imagine my chagrin to find the problem was something simple. At least it's simple once you understand how it's suppose to work.

First the basics, we open a port on the firewall, and configure Sendmail. Check out Denie's Blog for an explanation of the steps. Assuming you've got a running Sendmail / Dovecot server, here's the quick and dirty:

yum install -y cyrus-sasl
chkconfig saslauthd on
service saslauthd start
echo "mydomain.xyz RELAY" >> /etc/mail/access

All authentication must always be encrypted. Setup SSL by piggybacking on Dovecot:

mkdir -p /etc/pki/sendmail/{certs,private}
cd /etc/pki/sendmail
ln -s ../../dovecot/certs/dovecot.pem certs/sendmail.pem
ln -s ../../dovecot/private/dovecot.pem private/sendmail.pem

And changes to /etc/mail/sendmail.mc

dnl Authenticated send from mobile dnl
DAEMON_OPTIONS(`Port=1234, Name=MTA, M=Ea')dnl
dnl No anonymous logins (y) dnl
define(`confAUTH_OPTIONS', `A y')dnl
TRUST_AUTH_MECH(`LOGIN')dnl
define(`confAUTH_MECHANISMS', `LOGIN')dnl
define(`confCACERT_PATH',`/etc/pki/tls/certs/')dnl
define(`confCACERT',`/etc/pki/tls/certs/ca-bundle.crt')dnl
define(`confSERVER_CERT',`/etc/pki/sendmail/certs/sendmail.pem')dnl
define(`confSERVER_KEY',`/etc/pki/sendmail/private/sendmail.pem')dnl

Last step, restart Sendmail, test, and it was the test that I was messing up on. It always failed. To test, use telnet:

$ telnet 1.2.3.4 1234
Trying 1.2.3.4...
Connected
Escape character is '^]'.
220 ESMTP Sendmail 8.14.4/8.14.4; Thu, 24 Feb 2011 12:57:44 -0500
ehlo localhost
250-Hello pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-AUTH LOGIN
250-STARTTLS
250-DELIVERBY
250 HELP

First, we're looking for AUTH LOGIN. We need to send our username and password... but we have to send it using BASE64 encoding. This is security through obscurity, in the respect that anyone can crack BASE64. To change our clear text to the expected format, we use something like Ostermiller's Javascript utility. The next trick is that your username is the sending email address which has to be in the format of:
      unixname @ mydomain.xyz
...where the domain is the one we added for RELAY to /etc/mail/access. In the open telnet session:

AUTH LOGIN
334 VXNlcm5hbWU6
ZG9zz0BzzW5zzXIzzXM=
334 UGFzc3dvcmQ6
bWzzbSzzNQ==
235 2.0.0 OK Authenticated

We're in. Make sure to add the login name in the client in the same A@B.C formal.

We need to make a last change to force the rejection of clear text passwords. Back to the /etc/mail/sendmail.mc

dnl Only authenticate across SSL (p) dnl
define(`confAUTH_OPTIONS', `A p y')dnl

Add the "p" option, and restart Sendmail.