Monitoring Application Network Connections

Originally published January 7, 2018 @ 2:46 pm

A Java application on one of my servers has been causing problems by opening up too many network connections and not closing them quickly enough. I needed to be notified when the number of connections opened by a particular process exceeded a certain limit. I also needed a record of open connections over time. Below is a quick script I wrote for that.

There’re a couple of things I should point out. First, the options used with the netstat command:

netstat -tunap

The “p” is for “- -process”, which takes PID as argument. Also, I did not want to be notified more than once per day, so with every email alert the script touches a lock file and then compares the mtime on the file to the current time. If mtime is more than 24 x 60 x 60 seconds in the past (24 hours), the another email will be sent and timestamp on the lock file will be updated.

if [ $(echo $(( (`date +%s` - `stat -L --format %Y ${lockfile}`) < (24*60*60) ))) -eq 0 ]

This syntax will return 1 as long as the lock file is less than 86400 seconds old. The output will be 0 when the lock file is older than 24 hours, at which point another email alert may be sent. Also, note how the first letter of the “[p]rocess_string” is enclosed in square brackets. This is avoid matching “grep” in “ps -ef | grep” output. Finally, I use “grep -m1” to make sure I have only one PID. If you need to match multiple PIDs, you would need to use a “for” loop.

And here’s the script (also on GitHub):

#!/bin/bash
lim=1000
email="email@domain.com"
this_script=$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")
time_db=$(date +'%Y-%m-%d %H:%M:%S')
lockfile="/tmp/${this_script}.lock"
logfile="/var/tmp/${this_script}.log"
this_host=$(echo ${HOSTNAME} | awk -F'.' '{print $1}')

pid=$(ps -ef | grep -m1 "[p]rocess_string" | awk '{print $2}')
c=$(netstat -tunap | grep -c ${pid})

echo -e "${time_db}\t${this_host}\t${pid}\t${c}" >> "${logfile}"

notify() {
	echo "process_string PID ${pid} has opened ${c} TCP connections on ${HOSTNAME}" | \
	mailx -s "process_string network alert from ${HOSTNAME}" "${email}"
	touch "${lockfile}"
}

if [ ${c} -gt ${lim} ]
then
	if [ -f "${lockfile}" ]
	then
		if [ $(echo $(( (`date +%s` - `stat -L --format %Y ${lockfile}`) < (24*60*60) ))) -eq 0 ]
		then
			notify
		fi
	else
		notify
	fi
fi