more ssh log parsing and monitoring

Server SSH log files provides us information from simple system to critical system security message.

This is a continuation of parsing ssh log file blogged recently which can easily be found here. I have chosen for now, to look back of parsing ssh log files as openssh is one of the most useful, most powerful tool and widely used linux command on managing servers specially remote ones.

From previous blog entry, it was also covered an issue wherein we can generate these top lists of message details of a particular grep result that we can find from parsing /var/log/secure* log files.

Here are quick ways of doing it via CLI terminal. These examples could serve also as starting point to parse more search key strings not only from any ssh log files, but also to other daemon service log file such as dovecot, postfix, sendmail, apache, mysql, iptables, DNS, chilli, radius, voip applications, and more linux service log files.

Here goes.

How to generate top 10 list of most authenticated ssh logins by a user regardless of date and IP address based from /var/log/secure* log files?

One shot command:
# cat /var/log/secure* | grep Accepted | awk '{print $9 " " $11 }' | sort | uniq -c | sort -rn | head -10

Sample filtered output:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
153 vertito 192.168.100.3
124 vertito 192.168.100.25
56 vertito 192.168.100.18
41 vertito1 192.168.100.30
37 v 192.168.100.25
32 vertito2 192.168.100.3
6 vertito3 192.168.100.25
5 vertito5 192.168.100.10
2 vertito1 192.168.100.14
1 root 192.168.100.3
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Interpretation:
There were 154 successful ssh logins made by bash-enabled user account named vertito coming from 192.168.100.3 IP address, it was the highest in number based on the recent query.

Just add more parameter to show the results with corresponding dates like so:

We are only adding $1 and $2 , which are the month and the day of the month respectively.

# cat /var/log/secure* | grep Accepted | awk '{print $9 " " $11 " " $1$2}' | sort | uniq -c | sort -rn | head -10

You can actually create a bash script that does this one liner command and redirect the output to a file. From there, the bash script counts the output list from the command or from the generated file. If the count is more than a threshold, then further notification and action is done by sending email to server admins.

Linux commands comes in many variety and flexible ways specially when combined and redirected to or from another linux command.



Now, let's get the most authenticated IP address of any ssh user account.

How to generate top list of most commonly IP address on all successful logins from ssh default log file regardless of user names and dates?

# cat /var/log/secure* | grep Accepted | awk '{print $11 }' | sort | uniq -c | sort -rn

Sample results, modified to hide real IP address
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
186 192.168.200.3
168 192.168.200.25
56 192.168.200.18
41 192.168.200.30
5 192.168.200.10
2 192.168.200.5
2 192.168.200.14
1 192.168.200.17
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Interpretation:
There were 186 times of successful authenticated ssh logins from IP address 192.168.200.3, it was the most used IP address with successful ssh authentications.

Now, how about getting the top 10 of the most highest number of successful ssh authentications based from day of the month?

# cat /var/log/secure* | grep Accepted | awk '{print $1 " " $2 }' | uniq -c | sort -rn | head -10
Result:
~~~~~~~~~~~~~~~
62 Jun 10
33 Jun 23
29 Jun 25
19 Jun 26
18 Jun 8
18 Jun 1
16 Jul 9
13 Jun 12
13 Jun 11
12 Jun 7
~~~~~~~~~~~~~~~
Interpretation:
There were 62 successful ssh authentication attempts last June 10. It was the highest in number of most successful ssh logins based from /var/log/secure* log files.

From the above example, remember that we are only 'grep'-ing "Accepted" as a search key. Accepted refer to successful ssh logins whether by password authentication or private/public key authentications depending on your ssh server configs.

You can expand these queries to a more wider scope and also apply them to any other log files of your server with proper search key or filter words such as 'refused' , 'denied' and more. Hope to cover the other log files too sooner or later. These bash commands combination can also be converted to perl expressions as well, that can also be issued from command line terminal.

If this blog entry has been useful and/or informative to you in any way, you know what to do.

Goodluck!