Sunday, February 27, 2011

Mysterious Linux Permission Dots

Sometime mid last year, something strange happened: a dot appeared in the permission string of the Fedora distributions. I ask around, and no one knew where they came from, or what they were. Now I know, but first, let me show you what I'm talking about:
$ touch test.txt
$ ls -l test.txt
-rw-rw-r--. 1 doug doug 0 Feb 27 20:21 test.txt
It's hard to see, but its the eleventh character in the permission string:
1        Type of object, ex: d for dir or l for link
2-4     Permissions for owner
5-7     Permissions for group
8-10   Permissions for others
11      Mystery dot
Originally, I though this was an ext4 thing, but when I mount an ext3 under Fedora 13, I still get the dot.

Turns out, the mkfs commands have been modified to set ACLs on by default, and the dot is a place holder to represent an empty ACL. Previously, an empty ACL was not represented, as all ACLs are empty by default. Try this:
$ sudo tune2fs -l /dev/sda6 | grep options
Default mount options: user_xattr acl
$ setfacl -m u:apache:r test.txt
$ ls -l test.txt
-rw-rw-r--+ 1 doug doug 0 Feb 27 20:21 test.txt
By executing a setfacl command (as in set file ACL... ACL is pronounced like "ack ull") we change the dot to a plus, which tells us the ACL is no longer empty.
$ getfacl test.txt
# file: test.txt
# owner: doug
# group: doug
If we blank the ACL, the plus is gone, and the dot is back:
$ setfacl -b test.txt
$ ls -l test.txt
-rw-rw-r--. 1 doug doug 0 Feb 27 20:21 test.txt

Thursday, February 24, 2011

Authenticated ESMTP over SSL with Sendmail

I've been wanting to get this running, but kept running into the same problem. Imagine my chagrin to find the problem was something simple. At least it's simple once you understand how it's suppose to work.

First the basics, we open a port on the firewall, and configure Sendmail. Check out Denie's Blog for an explanation of the steps. Assuming you've got a running Sendmail / Dovecot server, here's the quick and dirty:
yum install -y cyrus-sasl
chkconfig saslauthd on
service saslauthd start
echo " RELAY" >> /etc/mail/access
All authentication must always be encrypted. Setup SSL by piggybacking on Dovecot:
mkdir -p /etc/pki/sendmail/{certs,private}
cd /etc/pki/sendmail
ln -s ../../dovecot/certs/dovecot.pem certs/sendmail.pem
ln -s ../../dovecot/private/dovecot.pem private/sendmail.pem
And changes to /etc/mail/
dnl Authenticated send from mobile dnl
DAEMON_OPTIONS(`Port=1234, Name=MTA, M=Ea')dnl
dnl No anonymous logins (y) dnl
define(`confAUTH_OPTIONS', `A y')dnl
define(`confAUTH_MECHANISMS', `LOGIN')dnl
Last step, restart Sendmail, test, and it was the test that I was messing up on. It always failed. To test, use telnet:
$ telnet 1234
Escape character is '^]'.
220 ESMTP Sendmail 8.14.4/8.14.4; Thu, 24 Feb 2011 12:57:44 -0500
ehlo localhost
250-Hello pleased to meet you
250 HELP
First, we're looking for AUTH LOGIN. We need to send our username and password... but we have to send it using BASE64 encoding. This is security through obscurity, in the respect that anyone can crack BASE64. To change our clear text to the expected format, we use something like Ostermiller's Javascript utility. The next trick is that your username is the sending email address which has to be in the format of:
      unixname @
...where the domain is the one we added for RELAY to /etc/mail/access. In the open telnet session:
334 VXNlcm5hbWU6
334 UGFzc3dvcmQ6
235 2.0.0 OK Authenticated
We're in. Make sure to add the login name in the client in the same A@B.C formal.

We need to make a last change to force the rejection of clear text passwords. Back to the /etc/mail/
dnl Only authenticate across SSL (p) dnl
define(`confAUTH_OPTIONS', `A p y')dnl
Add the "p" option, and restart Sendmail.

Fratelli Pinot Grigio

I worried over this one, because it was at the low end of my budget: which is dangerously low. Its not hard to find a bad wine for under $7, but for the price, this was a nice wine. A respectable wine in league with Bella Sera.

If a beginner needed an inexpensive wine for a fundtion, this would be a good choice. I'm inclined to recommend Bella Sera, but the label on this bottle looks classier... and some times that counts. An easy 5 out of 10.

Botter Verduzzo Prosecco

You know my wine motto: I don't drink blends. Well, this one was a Prosecco, so it doesn't count. And I can justify that...

Champagnes and spakling wines very seldom mention their veritals or vintages, so they are almost always a blend of something. For most sparklings, I complain about the over abundace of bubbles. But this one, was so different, truly! What was amazing to me was how this one "tasted" bubbly without being bubbly.

I really want to give this one an 8 of 10, but I think that the second bottle will be a 7.

Monday, February 21, 2011

MySQL Replication - Pt 3

And here's the stunnel config. The trick is that it has to be both ways, which is to say master has to be able to query slave and slave has to be able to query master. Thus, the same /etc/stunnel/stunnel.conf on both machines except for one value:
# logging
# setup

# ssl

# ssl
Remember, other_server is the only different on the two machines. Each point to the other. Even though the SSL certs are named the same, they are unique for each. The port listed in CHANGE TO MASTER will be the client accept (3307 in this example.)

To test, from both machines issue:
mysql -h --port 3307 -e "SHOW DATABASES;"
Add user names and passwords as needed.

One last note: The ports (3307 in this example) can be anything, but absolutely must be the same port number on master and all slaves.

Sunday, February 20, 2011

MySQL Replication - Pt 2

Through the wonders of technology, I've already been advised that my proof of concept is full of crap because:
* Replication does not provide automated fail over.
* After a fail over, there is no way to sync back to master.

Bullet one is true, but it is an excuse. Without replication, a catastrophic failure means getting a backup server online with data as old as my last backup or rsync. (And remember: rsync'ing MySQL is notoriously unreliable.) With replication, I'm within a few transactions. And don't bother me with your silliness about moving the RAID disks to a spare machine you had laying around.

Bullet two is only moderately true. The simple answer is to switch slave to read only, dump the database, transfer it to repaired master, and restore. Now we're synced, so we stop slave, and restore the original relationship. Of course, there is no way of knowing how long we'll be in read only.

As it turns out, we can make a few changes to deal with bullet two. As described in 5.0 16.3.6 we swap the relationship, allow the repaired system to sync as a slave. Once the repaired system has ingested the writes, we restore the relationship. Planned outage should be minutes.

On slave, add to /etc/my.cnf:
On master, add to /etc/my.cnf:
Effectively, we have added the ability for the slave to be master and master to be slave. If we add these at build time, the slave does not needed to be restarted at time of failure to become a master. Remember to add a replication user to the slave.

At time of master failure, we want to write to slave. From slave's MySQL console:
mysql> STOP SLAVE;
Switch front ends to write to slave. Once master is repaired issue the CHANGE TO MASTER command described in the previous post, modified as needed. Master will slave the missed writes. Once caught up, we fail back to master, and on slave issue START SLAVE.

So nah!

MySQL Replication

I've had a dozen people tell me that setting up a redundant MySQL server was just too difficult to be worth the effort. After all, what was the likelihood that the database server would ever go down? I mean... its not like tires on cars ever going flat, why would a computer ever break? Besides, the server is on a UPS.

Well... Bad news DBA's: I got it working in under an hour (not counting deploying the VMs and config'ing the base environment.) Its remarkably straight forward: the high altitude view is defined in RTFM 5.0, but of course, there are a few tricks.

Assuming a new cluster build, one will be a master, a second a slave. Start mysqld and verify functionality on both servers. On master, add to /etc/my.cnf:
server-id=2024561111 # some random value
On slave, add to /etc/my.cnf:
server-id=7035713343 # some random value
report-host=slaveXYZ # bonus: name of slave
Restart both servers. Verify

Log into master's MySQL console and add the replication user:
mysql> CREATE USER 'replicant' IDENTIFIED BY 'password';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'replicant';
Here's another trick: The replication is going to execute across the network in the clear, so ultimately, we want to push this through an SSL tunnel. With stunnel, we can specify the user as 'replicant'@'' (not localhost).

Determine the master's log coordinates: (yes, that's what its called)
We need to document the File and Position. Technically, the LOCK and UNLOCK are not needed since this is a new cluster. The right way to do this is to delay the UNLOCK until you verify slave is connected.

Log into slave's MySQL console and define the master:
-> MASTER_PORT=1234,
-> MASTER_USER='replicant',
-> MASTER_PASSWORD='password',
-> MASTER_LOG_FILE='mysql-bin.000001',
Another trick here: since we are going to use an SSL tunnel, we need a custom port value. Again, use the IP address, not localhost, to force it the TCP stack. Notice we used the file and position documented above.

The big gap in the documentation is its failure to tell you to actually start replication. On the slave:
  Slave_IO_Running: No
  Slave_SQL_Running: No
Did it work? Check the master:
Under Host you should see the "bonus" name specified in the slave's /etc/my.cnf.

And that's it. A big misconception seems to be that slave pulls from master, which is accurate, but not true. When master executes a write, it "pings" slave. If slave is up, it executes the pull. This could pose a problem if slave is DHCP. If slave is offline, it resyncs on start, which creates load on master. Depending on master's load, our systems could experience "lag".

Oh, and yes, I'm referencing 5.0, because that's what's included with RHEL5. My old copy of RHEL 6 Beta had 5.1 (I really should get a newer copy.) I glanced over the MySQL 5.5 docs, no big differences.

Blake Griffin Leaps Over a Kia for a Slam Dunk-

In LA, Sprite held a slam dunk contest at the Staple Center. I was impress by two shots. Wizard's Javale Mcgee did an impressive two basket, two ball, flying dunk. But the winner was Clippers rookie Blake Griffin leaping over the hood of a Kia, intercepting a basketball mid-air, and slamming it in while accompanied by a fifty person church choir.

Saturday, February 19, 2011

Stupidest Google Ad of the Week

I had just sent an email about an upcoming movie called "Appolo 18" and got this advert on my Gmail. "NASA Coupons" "Save now on NASA" What?

I had to click it. I got a coupon for Green Giant frozen vegetables. Or more correctly, I was given the chance to provide my email to someone for some reason, which may or may not have anything to do with NASA. Go ahead and click the picture above to see what you get... The more we click, the more it costs them.

Thursday, February 10, 2011

Interactive Organized Crime Map

Really cool interactive organized crime map on Wired. It seems smuggling Asian wood products is more profitable than human trafficking. Good to know.

Saturday, February 05, 2011

Go, Mark Kelly, Go

Astronaut Mark Kelly has the opportunity to command the last shuttle mission, yet there are people that don't approve. Not because he isn't qualified, but because they think that he should be by his wife Rep. Gabrielle Giffords side while she recovers from an assassination attempt by some drugged out whack job. Guess what people: your opinions on what he should do are worthless.

Is he qualified? Does he have his head on straight? Is he focused on the mission? Will his absence result in his wife's death? Those are the important questions... not should he do it.

Friday, February 04, 2011

Restrict Concurrent Remote Logins

I stumbled upon an interesting puzzle. I was asked to configure a system that would only allow a user to SSH from one remote address at a time, but allow multiple logins from that location. Furthermore, they can login from where ever they want, but never from two locations. Oh, and the restriction can't block the TTYs or Xterms.

I'd never heard of a scenario like that. There are lots of bells and whistles to lock down a system, but this one caught me off guard. After a few minutes searching the Interweb, I decided to whip out a hack:
# Restrict concurrent logins from multiple locations

MYT=`tty | sed "s~/dev/~~"`
MYL=`who | grep "$MYT" | awk '{print $NF}' | sed 's/[()]//g'`
MYC=`who | grep "$USER.*\..*\." | grep -vc "$MYL"`
if [ "$MYC" -gt 0 ]; then
  echo "Logged in on $MYT from $MYL"
  echo "Other remote locations:"
  who | grep "$USER.*\..*\." | grep -v $MYL | \
    awk '{print $NF}' | sed 's/[()]//g' | sort -u | xargs echo " "
  echo "Too many remote logins. Good bye."
  logger -p authpriv.warn "Killed remote login: $MYT $MYL"
  ps | grep -m 1 "$MYT" | awk '{print $1}' | xargs kill -9
  #fuser -k `tty`
Here's the flow:
  1. Determine our current TTY
  2. Get our remote address (client address)
  3. Are we logged in from another address...
      where the other address has two dots...
      and is not our own address
  4. If so, print all sorts of helpful information
      Note: comment "echo" lines in the wild
  5. Log the event
  6. Kick the bastard off
    fuser *should* have worked :(
Save it as /etc/profile.d/ and it will automatically be called after SSH authentication.

*** Update ***
A big mistake to avoid... Don't use the exit command in any script in the /etc/profile.d directory. This will cause the login process to exit, not the script.