The best way to secure a server is to limit its attack surface. The most reliable way to secure your server is to shut it down, unplug it and store it behind lock and key. Unfortunately, this is not practical and defeats the purpose to serving. We need to find a reasonable middle ground. While writing this post I swapped between my laptop running Slackware and a server which had FreeBSD, so I’ve included examples from both.
OpenSSH as a service
Most distributions come with OpenSSH as part of the base installation which is configured for a good balance between usability and security. Its up to the administrator to take the necessary steps to ensure the service is appropriately hardened before placing it out into production. Let’s take a look at the available means to refine the OpenSSH configuration with the goal of better security.
Checking the service
Before we start the configuration of OpenSSH, let’s first ensure that the service is running.
FreeBSD
In FreeBSD this is simple enough by querying the service init script:
# /etc/rc.d/sshd status sshd is running as pid 94980.
To go a little deeper we can see which ports are active and what daemons are controlling them by using the sockstat command with the -4 flag (for IPv4) to search for TCP/IP services.
drew@deimos:~ % sockstat -4 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS www httpd 96141 4 tcp4 *:80 *:* www httpd 96140 4 tcp4 *:80 *:* www httpd 96064 4 tcp4 *:80 *:* www httpd 96058 4 tcp4 *:80 *:* www httpd 96057 4 tcp4 *:80 *:* www httpd 96056 4 tcp4 *:80 *:* www httpd 96055 4 tcp4 *:80 *:* www httpd 96054 4 tcp4 *:80 *:* www httpd 96053 4 tcp4 *:80 *:* root httpd 96052 4 tcp4 *:80 *:* mysql mysqld 89508 10 tcp4 *:3306 *:* root sshd 76344 4 tcp4 *:22 *:* root sendmail 2657 4 tcp4 127.0.0.1:25 *:* root syslogd 1421 7 udp4 *:514 *:*
Among other services, we can see OpenSSH listening on port 22. Additionally, BSD has the more commonly known netstat command:
# netstat -na -f inet Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 *.80 *.* LISTEN tcp4 0 0 *.3306 *.* LISTEN tcp4 0 0 *.22 *.* LISTEN tcp4 0 0 127.0.0.1.25 *.* LISTEN udp4 0 0 *.514 *.*
Running the same command again, but with the inet6 argument we can see that OpenSSH is also listening on our IPv6 address.
# netstat -na -f inet6 Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp6 0 0 *.80 *.* LISTEN tcp6 0 0 *.22 *.* LISTEN udp6 0 0 *.514 *.*
Slackware
While Slackware doesn’t have the status argument in its out-of-the-box init script, it does give us more information with the netstat command. Not only does it give us our ports but also the PID and program associated with each port.
# netstat -plnt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:37 0.0.0.0:* LISTEN 711/inetd tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN 711/inetd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 717/sshd tcp6 0 0 :::22 :::* LISTEN 717/sshd
Whether you’re using FreeBSD or Linux, you can always use the tried and true process status command to see if your process is running, the PID number and confirm its path.
root@deimos:/var/log # ps aux | grep sshd root 94980 0.0 0.4 46876 4228 ?? Is 6:07PM 0:00.00 /usr/sbin/sshd
Restarting the service
Now that we’ve confirmed OpenSSH is running, let’s look at how to restart the service. When we make changes to the configuration we need to restart the service before those changes take effect.
FreeBSD
# /etc/rc.d/sshd stop Stopping sshd. # /etc/rc.d/sshd start Performing sanity check on sshd configuration. Starting sshd.
Slackware
# /etc/rc.d/rc.sshd stop # /etc/rc.d/rc.sshd start
Configuration
Now that we’ve verified the service is running and know how to cycle it; let’s get into the configuration.
Service port
We’ll jump right in with what is undoubtedly the most controversial topic in securing OpenSSH, the port number. Changing the service port from the standard 22 has increasingly become an accept mitigation technique. Security through obscurity as it were. The logic here is that if you use a non-standard port, you’ll greatly reduce the number of random attacks. Let’s examine this further.
As trite as the saying goes, the internet is still very much the wild west. The inter-connectivity, anonymity and the fact that it falls outside of national boundaries makes it a hard thing police. As you’d expect, in this sort of environment there is not shortage people looking to take advantage of others. Its no surprise then when you learn that the average server deals with hundreds of unsolicited service requests a day. Some of these requests are from the Googles and Bings of the world looking to update their web search database, others are from compromised systems looking for the next victim and sometimes these requests come from someone sitting behind a computer looking for holes to exploit. Interestingly enough, in most cases the last two are tied together.
The fact is there are machines that spend their day searching the net for servers to exploit. They are sometimes machines that have been explicitly setup to complete this task, but more often they are servers that have been previously compromised. These are called recon scanners and their job is to evaluate entire address spaces. Due to the vast number of IP addresses in these addressing spaces, these scanners often look for services on their standard ports. Each time they hit on a server which runs a service they are interested in, the scanner will attempt to authenticate and/or gather the service version number. In attempting to authenticate it will continue to try to login to the service using a dictionary of usernames and passwords (root/123456 anyone?), a technique called brute force intrusion. If the recon scanner successfully authenticates with the service or picks up a version of the service its looking for, it will populate the information into its database and move on. At this point the owner of the scanner can go through the logs and see what servers he now has credentials for or which servers are running services with known exploits.
In the above example the attacker is relying on weak passwords, unpatched services, or maybe an unknowing administrator. Even so, its important to note that running OpenSSH on a port other than 22 won’t make it any more secure though it may cover up for weak passwords, unpatched services or the fact that the administrator isn’t monitoring what’s going on. When choosing to change this option, its often best to look at the purpose of the server. Would I change this port if the service is accessed by clients? No. Would I change the port if the service was used by a small close-knit group of developers? Maybe.
For the sake of argument let’s assume that we’ve decided to change the port. Open the sshd configuration file in your favorite editor and search for ‘Port’. You’ll find a commented line specifying the port sshd will use to listen for connections. Uncomment the line and change the port to 2222.
Port 2222
Write your changes and let’s restart the daemon as per above.
Once restarted, running the service check again, we can confirm that the OpenSSH service is now listening on port 2222.
FreeBSD
# sockstat USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root sshd 86410 3 tcp6 *:2222 *:* root sshd 86410 4 tcp4 *:2222 *:*
Slackware
# netstat -plnt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:2222 0.0.0.0:* LISTEN 10488/sshd tcp6 0 0 :::2222 :::* LISTEN 10488/sshd
Listen address
By default OpenSSH will listen on all IP addresses. So, for example if this is a device that has interfaces on both a trusted and untrusted network, you can specify that OpenSSH only listen for incoming SSH requests on the trusted interface.
ListenAddress 172.16.0.10
The same can be done for an IPv6 address.
Protocol Version
For legacy purposes, you can still access OpenSSH Protocol Version 1. Unless you know of a reason you require it, ensure that your service only uses Version 2.
Protocol 2
Syslog
A vitally important, but often overlooked part of administration is logging. The syslogger is a means by which the daemon can be separated from the logging. The daemon controls what to log, while the syslogger handles there where. By doing so, the daemon doesn’t have to worry about management, storage and rotation of large log files. By default OpenSSH will send its logs directly to the syslogger, which is configured by default. Let’s take a quick look at how the syslogger is configured which will give us an idea of how OpenSSH logging operates. Open the file /etc/syslog.conf.
FreeBSD
*.err;kern.warning;auth.notice;mail.crit /dev/console *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log
Slackware
# Log anything 'info' or higher, but lower than 'warn'.
# Exclude authpriv, cron, mail, and news. These are logged elsewhere.
*.info;*.!warn;\
authpriv.none;cron.none;mail.none;news.none -/var/log/messages
# Log anything 'warn' or higher.
# Exclude authpriv, cron, mail, and news. These are logged elsewhere.
*.warn;\
authpriv.none;cron.none;mail.none;news.none -/var/log/syslog
# Debugging information is logged here.
*.=debug;kern.!=debug -/var/log/debug
# Logging for iptables
kern.=debug -/var/log/firewall.log
# Private authentication message logging:
authpriv.* -/var/log/secure
You will see the configuration is broken into two columns. On th left you’ll find what’s called the facility and level, and on the right the destination.
facility.level destination
What does this mean? In the fourth line of the FreeBSD example, it says that any messages generated for the auth or authpriv facility with the level of info be written to the /var/log/auth.log file. In a more complicated example, the fourth line tells us that any error messages (no matter the facility), kernel facility with the level of warning, authentication notices and mail critical level messages should be sent directly to the console. I bet at this point you’re thinking, “But where do these facilities and levels come from?” Our daemon configuration!
Switch back to your /etc/sshd_config file you will find a section called Logging. In here you can define the facility and log level generated by the OpenSSH service.
# Logging # obsoletes QuietMode and FascistLogging SyslogFacility AUTH LogLevel INFO
By knowing that we’re sending to the AUTH facility, we can cross reference this with our syslog.conf above and know that these messages are being sent to the file /var/log/auth.log.
I have merely skimmed the surface of logging here, its a subject worth pursuing in greater detail to ensure you have a properly secure system. Don’t forget that logs do nothing by themselves and need someone to review them. We’ll be visiting logs in the next section on log parsers.
Login Grace
While not directly affecting security, the setting LoginGraceTime determines how long a user can sit at the login prompt before being disconnected. Attackers have taking advantage of the LoginGraceTime setting in conjunction with MaxStartups to create a denial of service (DOS). Simply put, if there is a sufficiently long login grace, with a low max startups an attacker could saturate the server’s ability to answer OpenSSH requests. All this without having to login to the host. If you’re security conscious you can change the grace period to something shorter. The key here is to ensure you have a reasonable agreement between LoginGraceTime and MaxStartups.
LoginGraceTime 30
MaxStartups
Continuing from the point above, this setting determines how many connection attempts are permitted at one time. This is going to be defined by how many users access your server simultaneously, but be sure there is an agreement between this and LoginGraceTime
User Access
The fewer accounts allowed to login, the fewer opportunities for weak passwords. By default root is allowed to login, giving attackers a known valid username on your server. Once again open your sshd_config file and make the following modification.
PermitRootLogin no
Towards the bottom of the configuration you will find a setting called AllowUsers modify this setting so that only users who need access are granted it.
AllowUsers mike frank bill
Further to this there is a setting called AllowGroups which makes things easier. Instead of having to modify the configuration and restart the service when you want to modify users allow to use the service, you can create a group which is allowed access and thereafter modify memberships to it.
First create a group which to populate with users who will be granted SSH access.
FreeBSD
# pw groupadd sshusers # pw groupmod sshusers -m mike
Slackware
# groupadd –r sshusers # usermod –a –G sshusers mike
Once you’ve create the new group, add it to your configuration:
#AllowUsers mike frank bill AllowGroups sshusers
Password Tries
Simply put, how many times a user is allowed to retry their password before being disconnected. Obviously lowering this value will lower the efficiency of brute force attacks. From a log/monitor standpoint this makes it easier to identify brute force attempts as the attacker has to reestablish a connection more often. I’ve found three to be good number as it doesn’t inconvenience users.
MaxAuthTries 3
Key Authentication
One of the great features of OpenSSH is the ability authenticate with public-key cryptography. Due to its complexity, I won’t go into setting it up here, but it should be enough to say that by using key authentication you can forgo password authentication entirely. The advantage here is that with PasswordAuthentication disabled OpenSSH will no longer accept your typical username/password combinations thereby making brute force attacks futile. It goes without saying that this configuration takes a little more work as it required you to come up with a mechanism by which users can acquire a key to gain access to the system.
DenyHosts
Now that we’ve covered how to better secure your OpenSSH configuration, we’ll change gears and look at a commonly used log parser used to fend off brute force attackers. DenyHosts will continually review your logs for suspicious activity and block the offending IP addresses.
Here I’ve documented the installation and setup of the DenyHosts service on a Slackware.
Slackware package management appears to be largely unknown in the larger Linux community. While I’ve installed DenyHosts manually, there is a unofficial Slackware project called SlackBuilds which can be used to produce packages making upgrading and removal much easier.
Once you’ve downloaded the latest package from the DenyHosts homepage, extract it.
# tar xzvf DenyHosts-2.6.tar.gz
DenyHosts is written entirely in Python with a setup script to take care of most things for you.
# python ./setup.py install running install running build running build_py creating build creating build/lib *snip* copying plugins/README.contrib -> /usr/share/denyhosts/plugins copying LICENSE.txt -> /usr/share/denyhosts running install_egg_info Writing /usr/lib64/python2.7/site-packages/DenyHosts-2.6-py2.7.egg-info
With the application installed, let’s take a look at the contents.
# cd /usr/share/denyhosts/ # ls CHANGELOG.txt LICENSE.txt README.txt daemon-control-dist* denyhosts.cfg-dist plugins/ scripts/ setup.py
The first thing we need to do is take a copy of the denyhosts.cfg and daemon-control-dist sample files included in the installation.
# cp denyhosts.cfg-dist denyhosts.cfg # cp daemon-control-dist daemon-control
Next we customize the configuration so that it works with our system. Opening the denyhosts.cfg file one of the first things you’ll see is the reference to logging. This is the location we referenced in the syslogger files we looked at previously.
# Redhat or Fedora Core: SECURE_LOG = /var/log/secure # # Mandrake, FreeBSD or OpenBSD: #SECURE_LOG = /var/log/auth.log # # SuSE: #SECURE_LOG = /var/log/messages
Next look for the LOCK_FILE entry. Modify the line accordingly to correspond with Slackware convention.
#LOCK_FILE = /var/lock/subsys/denyhosts LOCK_FILE = /var/run/denyhosts.pid
With the daemon-control file we once again change the reference to the pid.
#DENYHOSTS_LOCK = "/var/lock/subsys/denyhosts" DENYHOSTS_LOCK = "/var/run/denyhosts.pid"
With the setup script completed and the configuration pieces in place, the last thing we need is a means to launch our service. So let’s create that script.
# vi /etc/rc.d/rc.denyhosts
Nothing fancy required, the following will do the job. (Note that sh/bin need to be reversed)
#!/sh/bin
dh_start() {
/usr/share/denyhosts/daemon-control start
}
dh_stop() {
/usr/share/denyhosts/daemon-control stop
}
dh_restart() {
dh_stop
sleep 1
dh_start
}
case "$1" in
'start')
dh_start
;;
'stop')
dh_stop
;;
'restart')
dh_restart
;;
*)
echo "usage $0 start|stop|restart"
esac
Ensure the script is executable.
# chmod +x rc.denyhosts
Now the fun part. Launch the script to have it parsing logs.
# /etc/rc.d/rc.denyhosts start starting DenyHosts: /usr/bin/env python /usr/bin/denyhosts.py --daemon --config=/usr/share/denyhosts/denyhosts.cfg
Let’s take a look at the denyhosts log to ensure everything looks healthy.
# tail /var/logs/denyhosts 2013-11-15 02:22:45,520 - denyhosts : INFO Processing log file (/var/log/secure) from offset (0) 2013-11-15 02:22:45,522 - denyhosts : INFO launching DenyHosts daemon (version 2.6)... 2013-11-15 02:22:45,524 - denyhosts : INFO DenyHosts daemon is now running, pid: 6697 2013-11-15 02:22:45,524 - denyhosts : INFO send daemon process a TERM signal to terminate cleanly 2013-11-15 02:22:45,524 - denyhosts : INFO eg. kill -TERM 6697 2013-11-15 02:22:45,572 - denyhosts : INFO monitoring log: /var/log/secure 2013-11-15 02:22:45,572 - denyhosts : INFO sync_time: 3600 2013-11-15 02:22:45,572 - denyhosts : INFO purging of /etc/hosts.deny is disabled 2013-11-15 02:22:45,572 - denyhosts : INFO denyhosts synchronization disabled
The denyhosts will now monitor your logs and before long you’ll see some new activity, like this happy chap from Serbia.
2013-11-16 02:14:20,907 - denyhosts : INFO new denied hosts: ['50.31.96.8']
Looking at your /etc/hosts.deny file you’ll see a new entry indicating that the address in questions has now been denied access to the OpenSSH service.
sshd: 50.31.96.8
The denyhosts.cfg file has many configuration options which allow you to tweak your parser including: the number of logins before denying an IP, how long to deny an IP, which services to deny the IP etc. Take some time and read though the file and customize accordingly. Also, to ensure you always have connectivity into your server, be sure to populate your IP address (if static) into the /etc/hosts.allow file. The hosts.allow entries will always trump hosts.deny.
Cautionary Note
While tools like DenyHosts and Fail2Ban are great for ensuring the safety of your server, keep in mind that they are only as good as the parsing logic they use.
# nc <your server> SSH-2.0-OpenSSH_5.3 Heya, I just put a bunch of stuff in your logs! :p Protocol mismatch.
And then in your logs:
Nov 26 21:17:05 sadhost sshd[24243]: Bad protocol version identification 'Heya, I just put a bunch of stuff in your logs! :p' from <attacker>
Just some food for thought.