Playing With Wire » SNMP http://www.playingwithwire.com The Internet Startup Blog Wed, 20 Jul 2011 18:45:29 +0000 en-US hourly 1 Monitoring Remote Hosts With Cacti, Part II http://www.playingwithwire.com/2008/08/monitoring-remote-hosts-with-cacti-part-ii/ http://www.playingwithwire.com/2008/08/monitoring-remote-hosts-with-cacti-part-ii/#comments Wed, 06 Aug 2008 18:25:42 +0000 http://www.playingwithwire.com/2008/08/monitoring-remote-hosts-with-cacti-part-ii/ Since we started writing here at Playing With Wire, we’ve managed to write two articles about Cacti. In January 2007 we introduced Cacti in the article “What’s Your Utilization, Kenneth?“. Six months later we wrote another article about how to monitor remote hosts with Cacti.

While the setup we described in these article worked out great, there were a few things that we didn’t really like about the setup:

  1. We connected with SSH from the Cacti server to the server we wanted to monitor (security issue).
  2. We were using regular SSH tunnels. While these are great, they do have a tendency to die (reliability issue).
  3. Due to security and portability, we wanted to isolate Cacti to a separate server (or VM).

1. Turning the tunneling around

The reason why we didn’t like to have the Cacti server connecting to the servers was simply that we needed one more user account on the remote servers. If these are production servers, it’s desirable to keep the publicly accessible user accounts to a minimum.

As it turned out, replacing the ‘-L’ with a ‘-R’ in the tunneling command turns the tunnel around. Instead of opening a port on the local machine, it opens a forwarded port on the remote server. By doing this, we can connect from the remote server to the Cacti server and still fulfilling the same purpose (but without creating an additional user on the remote server).

2. Creating more reliable tunnels

One of the major problems we were having with the setup was that the tunnels died for one reason or another. We initially solved this by writing a bash-loop that automatically reloaded the tunnel if that occurred. However, we were still experiencing some problems with dead tunnels.

The solution to the problem was autossh, a simple front-end to SSH that keeps the tunnel alive.

With autossh, we could simply launch the tunnels at boot-time on the remote server (in rc.local) without having to worrying about them dying. As we were implementing this on a number of servers, we wrote a small bash-script that launches autossh with the server-specific settings. The script looks like this:

#!/bin/sh
REMOTEPORT=2001
MONITORPORT=29001
/usr/bin/autossh -M $MONITORPORT -q -f -N -R
127.0.0.1:$REMOTEPORT:127.0.0.1:161 user@cacti.server.tld

This script creates a tunnel on port 2001 ($REMOTEPORT) on the Cacti host (cacti.server.tld) that goes to port 161 on the local machine (in this case, the server we want to monitor).

Isolate Cacti

In order to make our monitoring both more secure and portable, we felt that we wanted to isolate the monitoring to a separate Virtual Machine. This was easily done by creating a new VM under VMware Server. If you’re lazy, there are ready-to-use VMware images to download on the Cacti forum.

Bonus: Monitor several hosts with one tunnel

While reading the Cacti forum the other day, I ran across this article that talks about SNMP Proxies. By adding an extra entry in the SNMP config file, it’s possible for a single host to relay SNMP information about the other hosts on the network. This is very useful if you’re trying to monitor more than one host on the same network, as you don’t need one tunnel per server (beware that this creates a single-point-of-failure though).

Conclusion

After implementing these changes, we feel much more comfortable with our Cacti setup. Not only is it more reliable with more robust tunnels, but it’s also more secure.

The next Cacti-related task we will be looking at is to design custom plug-ins to monitor our own apps.

]]>
http://www.playingwithwire.com/2008/08/monitoring-remote-hosts-with-cacti-part-ii/feed/ 2