Thursday, August 2, 2007

ssh log parsing and monitoring

SSH is a remote login protocol used to login to a remote host for executing and managing remote machines. It is one of the most usable and known linux binary program ever, and it comes for free.

By default ssh installation, ssh open host port number 22 unless customized by user by editing /etc/ssh/ssh*config files. It also uses /var/log/secure as its default log file. This log file contains a lot of important authentication log details and system messages.

In addition from having rpm packages like denyhosts and logwatch, that scans and parses your server's ssh log files, I have compiled these quick and dirty brief examples below to parse /var/log/secure ssh log file and obtain more specific results.

All commands issued here would be done inside CLI terminal.

From /var/log/secure -

1. How to list out successful ssh login attempts?

# cat secure | grep 'Accepted'
# cat secure | grep 'Accepted' | awk '{print $1 " " $2 " " $3 " User: " $9 " " }'

2. How to list out successful ssh login attempts from sudo users?

# cat /var/log/secure | grep 'session opened for user root'
# cat /var/log/secure | grep 'session opened for user root' | awk '{print $1 " " $2 " " $3 " Sudo User: " $13 " " }'

3. If your server allows ssh logins by root. How to list out successful login attempts from root?

# cat /var/log/secure | grep 'Accepted password for root'
# cat /var/log/secure | grep 'Accepted password for root' | awk '{print $1 " " $2 " " $3 " User: " $9 " " }'

4. How to list out ssh login attempts from non-existing and unauthorized user accounts?

# cat /var/log/secure | grep 'Invalid user'

5. How to list out ssh login attempts by unauthorized ssh accounts?

# cat /var/log/secure | grep 'invalid user'

6. How to list out ssh login attempts by authorized ssh accounts with failed password?

# cat /var/log/secure | grep -v invalid | grep 'Failed password'

7. How to list out ssh login attempts by unauthorized ssh accounts with failed password?

# cat /var/log/secure | grep 'Failed password for invalid user'

8. How to list out refused ssh login attempts?

# cat /var/log/secure | grep 'refused'

9. How to list out denied ssh login attempts?

# cat /var/log/secure | grep -v cron | grep 'access denied\|Permission denied'

10. How to list out fatal and miscellaneous ssh session/restart notices ?

# cat /var/log/secure | grep ssh | grep 'Permission denied\|fatal\|error'

11. How to list out all system accounts that had their password changed by root?

# cat /var/log/secure | grep 'password changed'

12. How to list out all ssh service restarts?

# cat /var/log/secure | grep 'terminating'

13. How to count all ssh daemon service restarts?

# cat /var/log/secure | grep 'terminating' | wc -l

14. How to show all newly created system groups and system accounts?

# cat /var/log/secure | grep 'new group\|new user'

15. If you are using dovecot. How to show all failed authentication attempts from Dovecot ports?

# cat /var/log/secure | grep dovecot | grep 'user unknown'

16. How to show all deleted users and all removed groups?

# cat /var/log/secure | grep ssh | grep 'deleted user\|removed group'

17. How to show all changed user's home folder?

# cat /var/log/secure | grep change | grep 'home from'

18. How to show all successful system account expiration date changes?

# cat /var/log/secure | 'changed password expiry\|expiration from'

19. How to show all errors when changing group ID?

# cat /var/log/secure | grep 'Unable to change GID'

20. How to show all failed vsFTPd authentication attempts by authorized users?

# cat /var/log/secure | grep 'vsftpd:auth' | grep 'authentication failure'

21. How to show all failed vsFTPd authentication attempts by unknown users?

# cat /var/log/secure | grep 'vsftpd:auth' | grep 'user unknown'

22. How to show all failed proFTPd authentication attempts by unknown users?

# cat /var/log/secure | grep 'proftpd:auth' | grep 'no such user'

23. How to show all timed-out ssh login attempts?

# cat /var/log/secure | grep 'Timeout before authentication'

24. How to get get all authentication failures regardless of service name?

# cat /var/log/secure | grep 'authentication failure'

I guess this is it for now. Going along monitoring and managing ssh log files on regular basis is nice. SSH log file /var/log/secure shows different ssh log messages depending on how the service has been setup and configured.

Do take note the above examples show parsing ssh /var/log/secure alone as a single log file. It is possible to expand and cover all rotated ssh log files when issuing the above commands. This is possible by using file globbing approach, which as follow

# cat /var/log/secure*

With the above file glob filter, all matched rotated ssh log files would be included on the processes of executing the above command. Sample of rotated ssh log files are


and more, depends on your log rotation setup.

Do remember that this approach takes more processing time compared to parsing /var/log/secure log file alone. The new command would now look like

# cat /var/log/secure* | grep something

Now, the filtered results of the parsing the said log file could then be redirected to and as a file using linux command redirection. Like so:

# cat /var/log/secure* | grep someting > month1-week1-illegal-users.log

Archiving them for future references that is.

Why? What are the probable reasons for this?

Well, a few reasons for this is that you can actually reparse and reprocess the archived results by a shell scripts for more statistics that could be useful and handy. Ideas such as:

1. TOP N MOST denied/refused/active/timed-out ssh connection for this month/time?

Are they all legal and valid ssh connection events? Like if logged in ssh user is not on duty, how come somebody is using his shell account inside the building? Paranoid probabilities that can be drawn from here are really wide depending on the filtered results you can get.

Another one.
2. TOP N IP Address of MOST ssh denied attempt.

This data can be reparsed again. Using awk command, you can now blocked them permanently from the server via route and via /etc/hosts.deny . Useful?

3. TOP N IP address of MOST anything.

Who owns and manages the IP addy and IP block ? From which country of location? What time the event occurred? During sleeping time? Sunday? Maybe it was owned by a competitor with same business nature?

Perhaps informing the internet service provider of where the IP address came from is another way of helping the other admins to further implement necessary action policy, warnings and such. The info returned by dig, nslookup, and whois from that IP address is like a swiss knife too.

4. Creating a new bash script that dumps total counts of these iterations and occurrences would give you numbers. There are a lot to do with numbers. One, numbers can be an input values for MRTG, Nagios, or your own grapher mechanism. They can also be a number below a certain threshold for triggering an alarm and notice to send an alert emails or SMS message for you for your immediate attention and proactive response.

And more!

Just another ssh day.


Sign up for PayPal and start accepting credit card payments instantly.
ILoveTux - howtos and news | About | Contact | TOS | Policy