Dave Dribin wrote up a nice post about people breaking some of the first ‘s’ in ssh and how that annoys him and ways to get around it. Its really a good read and really hammers home the point of using ssh-agent, which Dave initially exposed me to. Being able to ssh around without typing my password each time was something I was missing back from when I had kerberized ssh back at UIUC. Er..Illinois.edu.
While I agree with Dave about the empty password thing, one thing you can do in your .ssh/authorized_keys file is lock down a key to a specific IP and to a specific command. So, while you may have a passwordless ssh access, you can somewhat mitigate the risk by locking down what they can can do. For example, with a key truncated because it is long:
from=”192.168.33.22″,command=”/usr/bin/uptime” ssh-rsa AAA…
With this addition in the authorized_keys section, a user with this key from 192.168.33.22 will only be able to execute /usr/bin/uptime. For more information on that and other options you can use, see the AUTHORIZED_KEYS FILE FORMAT section of the sshd(8) man page.