OpenSSH and Keychain for Systems Administrators

Background

This tutorial provides guidance on best practices and configuration of OpenSSH/Keychain, but also includes some important troubleshooting techniques for which documentation is somewhat lacking. These techniques took me several years to develop and I have tried to compile them here in one concise post so that others do not have to suffer through the arduous learning process.

Generally, it is considered sound security practice to create and use ssh key pairs which are password protected, but it can be an inconvenience because an administrator will be required to enter a password every time the key is used. Most administrators are aware that there is software to deal with this problem, but there seems to be a learning curve. This document attempts to provide the knowledge necessary to comfortably implement OpenSSH with Keychain.

The OpenSSH key agent can help by keeping the private key unlocked in memory, thereby preventing you from having to type the password over and over. The problem with the OpenSSH key agent, is that it only works for the currently active shell. For cron jobs and automation though, you will need Keychain. Keychain allows any shell owned by the same user to access and use the private key in memory. For a good background on Keychain please visit the Gentoo documentation.

This article is primarily written for Debian/Ubuntu & RHEL/Fedora distributions. While it should be easy enough to adapt the instructions here for others, your millage may vary. Finally, a note about the conclusions and best practices prescribed here. I do note claim to be an authority nor do I strive to prescribe these solutions with any authority. However, I have reached these conclusions through experience, research, due diligence and workmanlike character and publish them here for the benefit of the community. I think often systems administrators are afraid to commit to best practices such as these, so there is a severe shortage of good advice in an a practical format.

 

Best Practices

When setting up an SSH server, it is important to understand a few best practices. It is hard to find documentation which will give guidance on these critical elements, so I have created the following sections describing each decision. Also remember, decisions on security are an analysis of risk vs. efficiency at a high level and threat vs. vulnerability on the ground.

Passwords vs. Keys

Generally, stand alone SSH servers are protected by either key pairs or passwords. Passwords are often attacked using common dictionaries, while keys must be attacked by brute force. The protection these passwords/keys offer is measured in bits of entropy ((http://en.wikipedia.org/wiki/Password_strength)). Passwords, since they are fairly short and chosen by human beings, do not generally provide as much entropy as generated keys. For example, using all ASCII printable characters in a password provides about 6.555 bits of entropy per character ((http://en.wikipedia.org/wiki/Password_strength)). So, to get 65 bits of entropy, one would need a 10 character password with a good mix of upper case, lower case, numbers and special characters. A DSA key can be generated at 1024 bits, while a RSA key can be generated at 2048 or higher. Even if these pseudo random keys have only half the level of entropy as their bit length ((http://www.windowsecurity.com/articles/Ideal-to-Realized-Security-Assurance-Cryptographic-Keys-Part2.html)), they provide much more entropy than passwords.

Now, there are some mitigating factors. There is some delay between login prompts to an SSH server which provides some protection against brute force attacks, so passwords are not quite as insecure as some some might propose. Also, if one uses temporary lockouts and forced password changes, the vulnerability of passwords to brute force can be reduced greatly. But the fact remains keys provide more entropy with less policy guidance.

Conclusion: Use keys, there is generally entropy and they are easy to use.

 

Encrypted Keys vs. Unencrypted Keys

When your private/public key pairs are generated, you have the option of specifying a password. If you specify a password your private key will be protected with a symmetric encryption algorithm such as AES128 or 3DES. Remember this will only prevent a cracker from using your key immediately. The cracker will first have to successfully attack your password to decrypt the private key before it can be used to access your remote systems. This is useful in a scenario where a laptop is stolen and it is known that the private keys are lost or compromised. This buys the systems administrator some time to revoke the old keys and generate a new key pair before the cracker has time to decrypt the private key. It is important to understand that the symmetric encryption which protects the key pair is not immune to cracking and only buys time, the keys still need to be revoked if they are compromised.

Conclusion: Encrypted, there is plenty of software to get rid of the inconveniences

 

RSA vs. DSA

RSA is a trusted public/private key algorithm which was publicly revealed in 1978. A company called RSA Security formed around the patent and the algorithm was released into the public domain in 2000 ((http://en.wikipedia.org/wiki/Rsa)).

DSA is a private/public key algorithm which was developed by an ex NSA employee and adopted by the United States federal government for digital signatures under Federal Information Processing Standard 186 (FIPS 186) ((http://en.wikipedia.org/wiki/Digital_Signature_Algorithm)). The keys generated by this algorithm were not intended to be used for cryptography ((http://csrc.nist.gov/publications/fips/fips186-3/fips_186-3.pdf)), but were employed to do so in the mid 90s because RSA was covered under a patent. FIPS 186 specifies key lengths and the writers of ssh-keygen have decided to limit key length to the version 2 standard which is 1024 bits. This is a disadvantage for long usage periods ((http://www.keylength.com/en/4/))

From the OpenSSH 4.3 release notes, it appears that the decision was made to use FIPS 186-2 compliant keys. FIPS 186-2 also specifies that the DSA keys should not be used for any purpose other than signing. Since OpenSSH is using the keys for encryption which is not permitted by FIPS 186-2, it is difficult to understand the logic.

Reduce default key length for new DSA keys generated by ssh-keygen
back to 1024 bits. DSA is not specified for longer lengths and does
not fully benefit from simply making keys longer. As per FIPS 186-2
Change Notice 1, ssh-keygen will refuse to generate a new DSA key
smaller or larger than 1024 bits

Also, looking at the OpenSSH mailing list, this question was posted, but it appears there is little interest in increasing the key length for DSA keys

Conclusion: Use RSA based keys because ssh-keygen will allow longer key lengths

 

Key Length

Key length is easier to determine, there are algorithms to determine what is mathematically appropriate for the amount of time you will use the key. The following site provides an excellent overview of standards, key lengths, and time frame for which the key will be secure: keylength.com

Conclusion: At least 2048, it will last you until 2030

 

Root Login

There is debate over the prudence of allowing root login through ssh and as such there are several configuration options available. As of the this writing, the default configuration file in OpenSSH does permit root logins. A quick search of the OpenSSH mailing list does indicate that there are some who suggest changing the default behavior, though it doesn’t appear to have garnered much attention from the developers. Following the link cited by the poster, we find that there is a real danger of brute force attacks which would be especially applicable to a common account name such as root.

Conclusion: Do not permit root logins with passwords. This is especially true if ssh access is permitted from the Internet or if there is no intrusion detection system or log monitoring in place.

 

Login Banner

According to this SANS paper from 2007, login banners to provide legal strength should your server ever get hacked. Also, NIST recommends a login banner and provides a sample here.

Conclusion: Use a banner, it is simple to configure and easy to automate

 

Run on Off Port

This is security through obscurity, which does not directly strengthen your defense against being hacked. In the real world though, what this does do is reduce the number of port scans and automated attacks by orders of magnitude. This reduces the number of entries in your logs and also reduces the number of false positives.

In a small organization without sophisticated intrusion detection, this can be the difference between seeing the attack in the logs and missing it. In my experience, running on an off port is a simple filter to reduce the number of threats which gain visibility of your vulnerability. This is more useful on external servers than internal servers. The usefulness is greatly reduced on internal servers which should not show signs of brute force in the logs unless the attack is a real threat.

Conclusion: Run externally accessible servers on an off port. This is most useful in work groups and small companies where budget for sophisticated intrusion warning and detection systems are probably not in place.

 

Installation

In general, the OpenSSH client is installed on most distributions. The OpenSSH server and Keychain are not always installed by default, nor is Keychain, always available in the repository. Installation of all software is package based and does not have difficult dependencies.

Verify OpenSSH Packages are Installed

Fedora/Red Hat:
rpm -qa | grep openssh

Debian/Ubuntu:
dpkg -l | grep openssh

 

Install OpenSSH Packages

Fedora/Red Hat:
yum install openssh-clients openssh-server

Debian/Ubuntu:
apt-get install openssh-client openssh-server

 

Verify Keychain Package is Installed

Keychain makes using encrypted private keys dead simple. Keychain is a small script that will work in conjunction with ssh-agent to find and use your private keys from memory. At the time of this writing Keychain was not available in the RHEL or EPEL repositories, RPMForge must be used.

Fedora/Red Hat:
rpm -qa | grep keychain

Debian/Ubuntu:
dpkg -l | grep keychain

 

Install Keychain

Fedora/Red Hat:

yum install openssh-clients keychain

Debian/Ubuntu:

apt-get install openssh-client keychain

 

Then add a line similar to the following to your .bashrc or .profile. The following will automatically prompt for the password on your protected private keys the first time you log on. Also, if you have several keys protected with the same passphrase, Keychain will only prompt once.

keychain --nogui /root/.ssh/* &> /dev/null

 

Server Usage & Configuration

Using OpenSSH with Keys

First on the administrative box which will connect to the remote servers, generate a new private/public key pair dedicated. When prompted for a password, just hit enter (twice). I generally use the format of user@domain, but the pair can be named anything. This command creates the key pair where you can use it, in the .ssh directory.

ssh-keygen -t rsa -b 2048 -f /root/.ssh/[email protected]

 

The following two files will be created, they are the public and private key pair

[email protected]

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAqPW4d17e+G8VncDDA3du6YYjeH8dSrW8rNmHaMciHuS9zF/I
zS43xVEWuDarKQUo6J4H4uzhsEKC7RxYQLTLb8fmo8Y+S0X6QzNcqtlMD0toRWhd...

[email protected]

[email protected] AAAAB3NzaC1yc2EAAAADAQABAAABAQCo9bh3Xt74bxWdwMMDd27phiN4fx1Ktbys2YdoxyIe5L3MX8jNLjfFURa4NqspBSjongfi7OGwQoLtHFhAtMtvx+ajxj5LRfpDM1yq2UwPS2hFaF01bDKz0TiTOsal9jpGDmtYiRB4D7mte72esFY7EtGJRr5oduoJbkIfR/YMGdp4d+bjPzRlH+jP7C+ynAaKdCrI4bZ+1vuQX+PM9WKdgxuR2t0lEhW5kRDU2wX9uU1vh5H/gK5+RuYmhORlkdgNQqy0pk5NdZu99iX4z17oRVQ1NfjjlYm06h8+SNexvbD4Jr6RM81bttmUsFbRYPKbu3y92hxqT1fq1eUPOLP7 [email protected]

 

For convenience, when running scripts on remote machines, you will need to verify that key based authentication and remote root login are enabled. This is the default in OpenSSH and generally the default in most modern distributions. The following two directives should be specified.
cat /etc/ssh/sshd_config

PubkeyAuthentication yes
PermitRootLogin yes

 

Then you will need to distribute, the administrative server’s public key to the remotely controlled machines. Running the following commands over ssh to the remote server will do that.

echo "[email protected] AAAAB3NzaC1yc2EAAAADAQABAAABAQCo9bh3Xt74bxWdwMMDd27phiN4fx1Ktbys2YdoxyIe5L3MX8jNLjfFURa4NqspBSjongfi7OGwQoLtHFhAtMtvx+ajxj5LRfpDM1yq2UwPS2hFaF01bDKz0TiTOsal9jpGDmtYiRB4D7mte72esFY7EtGJRr5oduoJbkIfR/YMGdp4d+bjPzRlH+jP7C+ynAaKdCrI4bZ+1vuQX+PM9WKdgxuR2t0lEhW5kRDU2wX9uU1vh5H/gK5+RuYmhORlkdgNQqy0pk5NdZu99iX4z17oRVQ1NfjjlYm06h8+SNexvbD4Jr6RM81bttmUsFbRYPKbu3y92hxqT1fq1eUPOLP7 [email protected]" > /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

 

Using OpenSSH with Keychain

Follow the exact same instructions as above, but when prompted for a password, enter one (twice). This will encrypt your keys with AES128 on newer version of OpenSSH and 3DES on older versions.

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

 

Root Login

For more detail consult the man page for sshd_config, but essentially there are four main options: “yes”, “without-password”, “forced-commands-only” or “no” . In conjunction with Keychain and password protected private key it would be prudent to prohibit root login with passwords.

The option is poorly named, but will deny root from logging in with a password and essentially require a key (or some other form of login such as Kerberos). This offers the convenience of root based automation without the risk of brute force attacks on the root password

PermitRootLogin without-password

 

Login Banner

OpenSSH provides a pre-login banner with the Banner directive, which is simple to configure and automate during installation. Unix tradition dictates that, /etc/issue.net contains the banner text, but the FHS makes no mention of /etc/issue.net, only /etc/issue as an option component. Here, I will use the Unix tradition. The following directive goes in the /etc/ssh/sshd_config file
Banner /etc/issue.net

 

Generally, it’s not a bad idea to take a federal warning banner as an example and modify it for your own purposes, as the federal government has most likely covered all of their bases ((http://www.sans.org/reading_room/whitepapers/honors/cyberlaw-101-primer-laws-related-honeypot-deployments_1746)).

NIST Example Banner:
********************************************************************************
WARNING
********************************************************************************
This is a United States Government computer system that is for official use by
authorized users. Accessing and using this system constitutes consent to
monitoring, interception, retrieval, recording, reading, copying, searching or
capturing and disclosure of any information as to any information processed,
stored or manipulated within the system, including but not limited to
information stored locally on the hard drive or other media in use with this
unit internally or externally (e.g., floppy disks, tapes, CD-ROMs, PDA’s etc.)
by law enforcement and other personnel in conjunction with a report of improper
or unauthorized use. Unauthorized or improper use of this system is a violation
of Federal law and may be prosecuted resulting in criminal or administrative
penalties including fines and/or imprisonment. If criminal activity is
discovered, the information will be provided to the appropriate law enforcement
officials. Suspected access violations or rule infractions should be reported
to the Division Head, Regional Director or the Information Technology (IT)
Security Manager. The IT Security Manager can be reached on (202) XXX-XXXX.
********************************************************************************
WARNING
*******************************************************************************

 

Off Port Configuration

There are two main ways to configure which port OpenSSH runs on. The first method is to specify as a command line option during start up.

$SSHD -p3333 -f /etc/ssh/sshd_config_outside

The second is to specify the port in the configuration file /etc/ssh/sshd_config

Port 3333

 

Limit Users and Groups

The following directives can be added to /etc/ssh/sshd_config. It is important to note that these directives are greedy. If a list of users is permitted to AllowUsers, all other users will be denied by implication.

By User

AllowUsers scott sven lance
DenyUsers lotor lazerbeak

 

By Group

AllowGroups admins
DenyGroups dba

 

Temporary Lockout

This is useful especially with passwords. There are two main ways of locking out SSH, by IP address or by user. Neither of these is built into OpenSSH.

By IP Address

This script automatically adds the offending IP address to hosts.deny. Your OpenSSH server must support tcp wrappers. That can be checked with the following command.
ldd /usr/sbin/sshd| grep wrap

Download

• http://www.teamits.com/resources/maxlogins.html

By User

The simplest way to handle this is through PAM. A good starting point is pam_abl which has sane defaults.

Download

• http://www.kernel.org/pub/linux/libs/pam/modules.html

 

Further Security

There are many good online tutorials that give detailed configuration examples for locking down an ssh server, such as port knocking or iptables. Their work will not be duplicated here.

Good Links

• Rackspace SSH Security Guide
• SSH Lockdown
• Port Knocking

 

Client Usage, Configuration & Automation

Cron/Keychain Automation (Perl/Bash)

With Bash, writing scripts which will be called from Cron are easy. Simply, source the Keychain information. This will load all of the keychain environment variables and allow you to run any scripts from cron with no password prompt while still leaving your keys securely in memory.

source /root/.keychain/$HOSTNAME-sh

 

A native perl solution is to load the environment variables by iterating the information in the .keychain file

$HOST = `/bin/hostname`; chomp $HOST;
open(KEY, "/root/.keychain/${HOST}-sh") || die "Unable to get keychain information for $HOST\n   ";
while () {
    /([^=]+)=([^;]+)/;
    #print "env $1 = $2\n";
    $ENV{$1} = "$2";
}
close KEY;

 

Cron/Keychain Automation (any script)

This solution will allow you to run any script from Cron which leaving your decrypted private key in memory with Keychain.

*/5 * * * * (source ~/.keychain/`/bin/hostname`-sh; ssh [email protected] anycommand)

 

Auto-Accept Host Keys

Sometimes with doing mass ssh in a loop, it is useful to automatically accept host keys from known good machines. The following will automatically accept a server’s host key without prompting the user.

ssh -oStrictHostKeyChecking=no [email protected]

 

Socks Proxy

The advantage of this method is it allows access to any port on the remote side. This can be useful for a systems administrator who needs to connect to web based remote administration software which is only accessible from the local host. This is also useful for proxying to other hosts from a jump box.

ssh -D 8080 server.example.com

 

Notice the HP system management page in the background. It is connected to on the default port of 2381 which is only accessible from the localhost (from remote server to it’s self), but with the ssh SOCKS proxy in place, it can be securely managed remotely with the same port number.

 

TCP Forwarding

This allows static proxying of specific TCP ports over an ssh connection. This avoids having to configure your browser for a SOCKS proxy and works with any TCP connection.

ssh -L 2381:localhost:2381 [email protected]

 

Notice the HP system management page. It is connected to on the default port of 2381 which is only accessible from the localhost (from remote server to it’s self), but with the ssh TCP forwarding in place, it can be securely managed remotely by connecting to the localhost (systems administrator’s machine) with the same port number.

 

X11 Forwarding

This allows proxying of X11 over an ssh connection. This allows the remote administrator to run any X11 based application remotely. The disadvantage of this method is slow performance, especially over a VPN or a slow link to the server.

ssh -X [email protected]
firefox

 

Notice the HP system management page. It is connected to on the default port of 2381 which is only accessible from the localhost (from remote server to it’s self), but the browser is running on the remote host, it can be securely managed remotely by connecting to the localhost (remote server) as if the administrator where sitting at the graphical console.

 

 

Troubleshooting

These are just a few lessons learned and troubleshooting techniques to get started.

SSH Connection Problems

Often when running large numbers of ssh connection with automated infrastructure, at some point, there will be connection problems. The following options can mitigate connection problems in automated scripts. These options can be set explicitely on the command line, in ~/.ssh_config or /etc/ssh/ssh_config

ssh -o ConnectionAttempts=3 ConnectTimeout=3 server.example.com

 

Making and Testing Configuration Changes Live

This is useful when making changes you are unsure of, without getting disconnected from a remote server. A simple solution is to fire up a second server in the foreground on a different port with debugging enabled. You must specify the full path when launching the server. It may be necessary to stop a hosts based firewall, such as iptables, or make an exception while doing this kind of testing. Also, some servers may disconnect after one connection, while others may require Ctr-C to kill the process after wards.

/usr/sbin/sshd -p 2222 -v

 

Then connect from the client with debugging on. This will give you debugging from both sides and should solve any configuration or permission errors you might run into.

/usr/sbin/sshd -p 2222 -v server.example.com

 

Delay When Logging On

This can often be caused by DNS. Make sure the admin box has forward and reverse DNS in place or it might slow down connections. Also, double check that all of the name servers listed in /etc/resolv.conf are working right. From the client run the following using each DNS server listed.

Example:

host admin.example.com dns1.example.com
host admin.example.com dns1.example.com

 

OpenSSH Client Options with Rsync

Controlling ssh options with Rsync can be done with a shell variable or with a command line option. The first method works better when shell scripting because because there is no need to worry about escaping double quotes. I have found scenarios where it is impossible to pass the correct options to OpenSSH through the -e method.

 

Shell Variable

In general it is better to avoid quoting problems

export RSYNC_RSH="ssh -o ConnectTimeout=${ssh_timeout} -o ConnectionAttempts=3"
/usr/bin/rsync -av /src /dst

Command Line

/usr/bin/rsync -av -e "ssh -o ConnectionAttempts=3" /src /dst

 

Scripting Techniques for OpenSSH and Bash

Do not pass overly complicated commands to the remote machine. Try to implement all remote functionality with a script because quoting and escape characters can be a bear. For example, if you want to do a sed find/replace on the remote machine, do it in a script.

Instead of:

ssh [email protected] "sed -i -e 's/\$HOSTNAME/\$HOSTNAME-test/' /root/test.txt"

Write a script which takes a parameter

#!/bin/bash
sed -i -e 's/$1/$1-test/' $2

Then run

ssh [email protected] "/root/bin/replace.sh server1.example.com /root/test.txt"

 

SFTP Subsystem & Umask (Updated 01/2011)

This problem has gone away with recent versions of open ssh: Bug 1229. The umask will now be read from the user’s profile. There is now a -u option for sftp-server if the default umask needs to be changed.

In older versions of OpenSSH/SFTP, there is no easy way to let the user specify their own umask. OpenSSH and the SFTP subsystem will not obey the system umask, user umask, nor is there a way to specify one in the sshd_config file. On top of that the default umask for OpenSSH is 022.

If you have multiple web developers on a system making changes through SFTP, they will not be able to modify each other’s files. This is good for security and bad for usability. There is a work around. You can set the umask before calling the sftp subsystem or you can wrap the sftp subsystem in a simple script which modifies the umask.

Simple

If all you need to do is set the umask so that developers can colaborate, use this method. The following will set a slightly more liberal umask, then call the sftp subsystem. This combined with set gid on the working directories of a web server should allow developer interaction in the document root

Default:

Subsystem sftp /usr/libexec/openssh/sftp-server

 

Patched:

Subsystem sftp /bin/sh -c ‘umask 0002; /usr/libexec/openssh/sftp-server’

 

Here is snapshot of what the sftp subsytem looks like when running. Notice how it is a subprocess of sshd, this is what allows us to intercept and set the umask

├─sshd─┬─sshd───bash───pstree
│ └─sshd───scp

 

Advanced

Creating a wrapper script can allow you to add extra auditing and set the umask.

vim /usr/local/bin/sftp-server.sh

 

#!/bin/bash

umask 002
eval /usr/libexec/openssh/sftp-server

 

Then modify the OpenSSH Daemon configuration file to call this new script instead of directly calling /usr/libexec/openssh/sftp-server directly

vim /etc/ssh/sshd_config

 

Change the Subsystem directive to reflect the new wrapper script

Subsystem sftp /usr/local/bin/sftp-server.sh

 

Automation and Max Startups

Sometimes when controlling jobs from cron, several or even hundreds will execute at the exact same second. This can cause an ssh exchange key exchange error and disconnect the client. This can be alleviated by increasing the MaxStartups directive: For further information and guidelines, see this article: Systems Administrator’s Lab: OpenSSH MaxStartups

 

Force Disconnect

Sometimes when network connectivity is lost, the ssh client will simply lock up and no combination of normal Unix key commands will disconnect it. The following will work, type these keys in sequence. The actual command ssh client command is the last two keys, but they must always follow a new line.

ENTER
~
.

 

Links

The following is a good set of links for further reading.

• Gentoo Keychain Guide
• Rackspace Guide to Locking Down SSH
• IBM Developer SSH 1, IBM Developer SSH 2
• Digital Signature Algorithm
• RSA Algorithm
• Distributed.net Projects Breaking Encryption
• Ubuntu Guide
• GPU Cracking