When we launched YippieMove we quickly realized that we needed a reliable 3rd party that could ping our servers from several locations across the globe to ensure that we were not experiencing any problems with the access to our site. As we are quite tech-savvy here at WireLoad, we had a hard time justifying paying more than a few bucks per months for a service like this, since the service is so easy to write (we actually did write our own uptime-monitor with alerts a few years back using Curl, Crontab and some other tools, but would rather outsource this service).

So the search began. We required a few thing for this service:

• Several servers across the globe that ping our servers.
• Cheap. Preferably free (we don’t mind some ads).
• Decent statistics showing response-times etc.
• Reliable alert system by e-mail (luckily most US Cell providers allow you to send email to your phone, using your-number@theirdomain.com.)
• Must allow monitoring of both SSL and non-SSL servers.
• A minimum of 4 monitors (we needed to monitor playingwithwire.com, wireload.net, yippiemove.com [with and without SSL]), but it would also be great if we could monitor our mail-server.
• The more frequent the pings the better.
• No back-links required.

One of the most impressive sites we found was Pingdom, a small Swedish firm that is trusted by companies such as IBM, Loopt and Twitter (wow, they must spend more bandwidth on alerts than pings with Twitter for sure). What we really liked about Pingdom was the general look and feel of their site. It feels fresh, responsive and reliable. The pricing is definitely within reason: they charge $9.95 for their Basic plan, which includes 5 checks and 20 SMS.

The next site we stumbled upon was SiteUptime. The site has a decent look and feel (but does not come close to Pingdom). After examining their pricing, we realized that we needed their Advanced plan, since none of their lower plans allowed SSL monitoring. The price for this plan is $10 per month. While their site and visualization does not come close to Pingdom, they do give you 10 monitors, as apposed to 5 monitors with Pingdom, with their Advanced plan.

Another site we found was Pingability. The general look and feel of the site is OK, but the service offered was not great. The free plan requires a back-link (which we think is unacceptable for a professional site). At the same time the premium service, for $9.95, only offers one monitor.

Next up for review is Wormly. Priced at $9 per month, their Bronze-plan seems to be a reasonable alternative. The plan includes 5 monitors and they ping your server 5 times every 5 minutes, which is good enough. Unfortunately there’s a big ‘but’ — no SSL monitoring (at least as far as we can tell). That’s a deal-breaker. To Wormly’s defense though, they do offer something that sets them apart from the competition, namely the ‘Server Health Monitor.’ This service is something similar to Cacti (it definitely looks RRDTool-based), that visualizes server-load. However, they will probably have a hard time selling this service to security-concerned organizations, as they require a monitoring-client to be installed on the server (it’s hard to get this data otherwise).

Basicstate is the final service we will cover in this article. A lot can be said about Basicstate’s web design (it’s _really_ bad). However, they do offer a very competitive service. They ping every 15 minutes and allows you monitor as many sites as you want (including SSL). While it might not be a very pleasing site to browse, they do offer sufficient statistics (with graphs) on their site. In addition to that, they also send you daily reports about all your monitored sites (with time data for dns, connect, request, ttfb, ttlb). The only drawback we discovered with Basicstate is that you cannot monitor the same domain-name with SSL and non-SSL (sub-domains is fine though). This may or may not be an issue for you.

The verdict? We settled for Basicstate. Later on, as we grow, we might consider switching to Pingdom. We’re happy with Basicstate for now. Although we did experiencing some false alerts, the guy who runs the site (I assume), Spenser, did a great job on providing an in-depth explanation to the alerts by email. So if you’re on a tight budget, Basicstate is our recommendation. If you have more money to spend, go for Pingdom.

While the setup we described in these article worked out great, there were a few things that we didn’t really like about the setup:

1. We connected with SSH from the Cacti server to the server we wanted to monitor (security issue).
2. We were using regular SSH tunnels. While these are great, they do have a tendency to die (reliability issue).
3. Due to security and portability, we wanted to isolate Cacti to a separate server (or VM).

1. Turning the tunneling around

The reason why we didn’t like to have the Cacti server connecting to the servers was simply that we needed one more user account on the remote servers. If these are production servers, it’s desirable to keep the publicly accessible user accounts to a minimum.

As it turned out, replacing the ‘-L’ with a ‘-R’ in the tunneling command turns the tunnel around. Instead of opening a port on the local machine, it opens a forwarded port on the remote server. By doing this, we can connect from the remote server to the Cacti server and still fulfilling the same purpose (but without creating an additional user on the remote server).

2. Creating more reliable tunnels

One of the major problems we were having with the setup was that the tunnels died for one reason or another. We initially solved this by writing a bash-loop that automatically reloaded the tunnel if that occurred. However, we were still experiencing some problems with dead tunnels.

The solution to the problem was autossh, a simple front-end to SSH that keeps the tunnel alive.

With autossh, we could simply launch the tunnels at boot-time on the remote server (in rc.local) without having to worrying about them dying. As we were implementing this on a number of servers, we wrote a small bash-script that launches autossh with the server-specific settings. The script looks like this:

#!/bin/sh
REMOTEPORT=2001
MONITORPORT=29001
/usr/bin/autossh -M $MONITORPORT -q -f -N -R
127.0.0.1:$REMOTEPORT:127.0.0.1:161 user@cacti.server.tld

This script creates a tunnel on port 2001 ($REMOTEPORT) on the Cacti host (cacti.server.tld) that goes to port 161 on the local machine (in this case, the server we want to monitor).

Isolate Cacti

In order to make our monitoring both more secure and portable, we felt that we wanted to isolate the monitoring to a separate Virtual Machine. This was easily done by creating a new VM under VMware Server. If you’re lazy, there are ready-to-use VMware images to download on the Cacti forum.

Bonus: Monitor several hosts with one tunnel

While reading the Cacti forum the other day, I ran across this article that talks about SNMP Proxies. By adding an extra entry in the SNMP config file, it’s possible for a single host to relay SNMP information about the other hosts on the network. This is very useful if you’re trying to monitor more than one host on the same network, as you don’t need one tunnel per server (beware that this creates a single-point-of-failure though).

Conclusion

After implementing these changes, we feel much more comfortable with our Cacti setup. Not only is it more reliable with more robust tunnels, but it’s also more secure.

The next Cacti-related task we will be looking at is to design custom plug-ins to monitor our own apps.

In the scope of this article, we assume that you already have one Linux or Unix server configured with Cacti running. Moreover, we also assume that you have another remote server that you want to monitor (why else would you read this article?).

First we start by setting up the remote server. The first thing we need to do is to set up Net-SNMP. If you’re running FreeBSD, chances are that you already have this installed. If so, all you need to do is to change your community string and set up the daemon to bind on port 161/tcp instead of 161/udp. To do this, change/add the following lines in your snmpd.conf (/usr/local/share/snmp/snmpd.conf in FreeBSD):

com2sec local localhost public
agentaddress tcp:161

Once this is done, go on and restart the daemon.

To test that the SNMP daemon is working properly, try to run the following command:

# snmpwalk -v 1 -c public tcp:localhost:161

If your screen gets flooded with information, it worked. If not, please look over your log-files to find out what went wrong.

Next we need to create a secure user which we will be able to use to login from the Cacti-machine onto the remote machine. To maximize security, we suggest that that user has ‘nologin’ as shell and uses public key authentication instead of password. However, we will not cover how to create this user in this guide. Ask Google for help if you need it.

Repeat this for all hosts you want to remotely monitor.

That’s it for the remote server, now let’s move on to the Cacti host.

First we want to create a script that sets up the tunnel to the remote server. We suggest that you create a new user that will be running these tunnels (i.e. snmp). In order to make it easier to manage the tunnel (or tunnels if you have several hosts), we will create a bash-script that initializes the tunnel(s). In the home directory of the the user you created, create a file called tunnels.sh with the following contents:

#!/bin/sh
rm /home/snmp/tunnel.log

# server1.xyz.net
ssh -N -L 16000:127.0.0.1:161 snmp@server1.xyz.net >> /home/snmp/tunnel.log &

# server2.xyz.net
ssh -N -L 16001:127.0.0.1:161 snmp@server2.xyz.net >> /home/snmp/tunnel.log &

Note that this initalize two tunnels, one to server 1 (on port 16000) and one to server 2 (on port 16001. Also, don’t forget to chmod the file so that you can execute it, by typing chmod +x tunnels.sh.

Next we want to start up the tunnels using the snmp-user we created earlier. To do this run:

#sudo -u snmp /home/snmp/tunnels.sh

If everything went fine, you should now have two tunnels running; one on port 16000 and one on port 16001. Now let’s test the tunnels before we move on to Cacti.

# snmpwalk -v 1 -c public tcp:localhost:16000
# snmpwalk -v 1 -c public tcp:localhost:16001

This should hopefully give you the the same output as you previously received when executing snmpwalk locally on the remote hosts. If this went well, all you need to do now is to add the hosts to Cacti.

First you need to log into Cacti with an administrative account. Then got to “Create Device.” In the Create Device field, as shown in the screenshot bellow.

Description: server1.xyz.net
Hostname: tcp:127.0.0.1
Host Template: ucd/net SNMP Host
SNMP Community: public
SNMP Version: Version 1
SNMP Port: 16000

After you’ve filled out the proper data, hit ‘Create.’ At the next page, just select the data you want to graph, and then hit ‘Next.’

That should be all you need to graph remote hosts. Now you may want to go ahead and add the host to a tree so that you can display it in the ‘Graph’ tab.

           

At the time of the article, I installed Cacti for one of the organizations that I administrate the IT infrastructure for. Not only did I get a better idea of the utilization of bandwidth and hardware on the servers, but I could also see how much CPU resources the workstations were consuming. Although I knew that the CPU usage on the servers was quite low, I didn’t anticipate that the CPU usage on the workstations was quite as low as it was.

The organization is quite a typical office environment with 20-some workstations running mainly our own software plus web, e-mail, word-processing and spreadsheet applications. The hardware is quite modern with CPUs ranging from 1.8 Ghz to 2.7 Ghz Celerons and RAM between 256 MB to 512 MB. All workstations are running Windows 2000 Professional. Before I installed Cacti, I thought that the CPU usage during day use would average maybe 30-40%, with some significant peaks pushing up the average. However, I was quite surprised to find out how wrong my estimate was. It turned out that the average CPU usage on these workstations was less then 10% for all machines, and less than 5% for most of the machines with only few significant spikes. It’s true that Cacti only polls information from the workstations every 5th minute, but it should still give a quite accurate value these passed months as I’ve been running it.

With this data on hand it’s quite obvious that we’ve been over-investing in hardware for the workstations. Even though I would rather overestimate a little bit than underestimate, it seems my estimates were far too high. Even so, when purchasing new workstations, we’ve pretty much bought the cheapest Celeron available from the major PC vendors, so maybe it would have been hard to adjust the purchases even with these figures on hand.

After some thinking I came up with three possible ways to deal with this problem:
1. Ignore it
I guess this is what most companies does. Maybe the feeling of being ‘future-proof’ is valued more than the fact that you have a lot of idle-time. The benefit of doing this is that you have modern hardware that is less likely to break than old hardware.

2. Buy used hardware
Most people in power would be scared of just thinking about this. However, since there is no really new low-end hardware available, this appears to be the only way. The first problem you will be facing is probably to find uniform hardware. As an administrator, you know how much easier it is to administrate 20 identical workstations, rather than 20 different workstations, both in terms of drivers and hardware maintenance.

The second problem I came thinking of is the reliability problem. Obviously 5-10 years old hardware is more likely to break than brand new hardware, and if it does, there is no warranty to cover it. However, if you buy used hardware your budget will likely afford you to buy a couple of replacement computers.

There are also security implications of buying used computers. Every modern company with an intelligent IT staff is really concerned about security, both software and hardware. If you buy used hardware, there is a chance that it might be compromised (hardware sniffers etc.) I guess the only way to deal with this problem is to carefully physically inspect all the hardware you purchase.

If you or your company do choose to buy used hardware, there are plenty of sources to do so. One of the more interesting pages I found was a company in Australia, called Green PC, which sells a variety of computers and peripherals for a reasonable price.

3. Donate idle-time (to internal or external use)
With the rise of clusters, distributed computing and virtualization, there are today plenty of ways to put idle-time to good use. One of the more famous projects that deal with this is Folding@Home, which is a project at Stanford University that uses the participants’ idle-CPU/GPU time to do medical research. More recently a project at Berkeley called BOINC created a program that lets the user choose between a variety of distributed computing projects within the same application. By participating in such project, the company will create positive publicity (if the participation is significant).

If your company isn’t interested in donating idle-time to charity/research, they might still be able to use the idle-time. If all your workstations are connected with a high-speed connection (preferably gigabit), you might be able to use the computers in a virtualization environment. However, this is doable in theory, but I don’t know how well this would work in reality. Another alternative might be to use the idle-time to internal computations. If your company is in the software-business, distributed compiling might be one way to use the CPUs more efficiently. If this is not interesting, there are plenty of distributed computing solutions that might be used for intranets for various calculations that the company might else use a 3rd party company to compute.

Hopefully you have a better idea of how you can use your idle-time more efficiently, but you should be careful though. Some people argue that today’s computers are not built to run at 100% utilization 24/7. This is a very valid point, since neither the components on the motherboard, nor the fans is likely to stand a 100% utilization for very long without breaking. Therefore it is recommended to try to find an optimal distribution algorithm that spreads the calculation over the nodes without pushing the individual workstations until they break. I have to admit that I don’t have any available data on how the life-time of the workstations will be affected by running these type of softwares, but I would guess that it will have some impact on the life-time.

To round up this article, I would like to discuss one question that is highly relevant: “Why are there no low-end, cheap computers available?”

If you go to Dell or HP’s homepage and look for their cheapest ‘office’ hardware, it’s still far more than what is required for most office use. So why is it this way? As I see it, there are several reasons for this, involving both the software and hardware manufacturers in a mutual effort to stimulate sales. Obviously, the hardware manufacturers want us to replace our computers as often as possible, since this is how they make their profits. The software manufacturers on the other hand, want to sell new versions of their softwares by implementing new fancy features, that is unlikely to add to productivity, but requires hardware upgrades to run properly.

Let’s say you’re onboard with my ideas, and that you decide to look for cheaper hardware, but still feel like used hardware is too risky. One possibility might be to go for some kind of ITX-solution. These comes with a less powerful CPU, and often includes everything you need for desktop usage, but costs less than regular computers. One benefit of using ITX boxes is that they are very tiny and light, which makes them cheap to store and ship internally.

             

This happened to us a couple of weeks ago. We were looking for a way to keep track of the server utilization. We needed to find out the usage during peak hours, to determine if it was time to upgrade to new hardware or buy another server soon or not.

After a bunch of hours googling and searching the Gentoo forum (guys, we’re still friends, right?), I found a thread that discussed this exact problem. After looking through a couple of the applications listed, I found it. The answer to all my problems was spelled Cacti.

Cacti is simply a web-based SNMP-client that uses RRDTool to generate nice graphs. Sounds quite simple, right? Why am I so impressed and excited? Because Cacti is really SNMP made easy.

I’ve been looking at similar solutions before, but everything I found felt very Beta / “Hack it to make it work on your system.” Cacti on the other hand was really easy to install and configure. It probably took me about 10 minutes to configure it for my needs, and then another 10 minutes to get a local SNMP daemon to run (even though this is not necessary).

Cacti requires the following:
- Apache (might work with other webservers)
- PHP
- RRDTool
- MySQL
- A crontab-job

After configuring the database-settings for Cacti, you just add a crontab to execute a given PHP-page (poller.php) on a given interval (5 minutes), and you’re set. Now you can start adding your other SNMP-enabled devices to your Cacti page.

One of the demo-sites listed on Cacti.net

So let’s say you have this cool device that supports SNMP, but you don’t really feel like writing a custom template for the device. Well, just head over to the Cacti forum and search for the device. Chances are you’ll find that someone already wrote a template for the device. Take a look at the forum and you’ll find that people have written templates for all kinds of random stuff.

Things that we use Cacti to monitor:
- CPU usage
- Memory usage
- Network usage (both servers, routers and APs)
- Individual daemons (Apache, MySQL etc.)
- Laser printers (to monitor toner level)
- UPSes (with a plugin to get info from NUT)

Some of our graphs:

CPU Usage when PlayingWithWire.com got Slashdotted

Bandwidth status on one of our routers

The status of my HP Color LaserJet

It’s very convenient to just browse into Cacti to get a quick overview of your network/server utilization. In addition to that you can also select what specific time-span (daily, weekly, monthly etc.) you want to see. I love it.

If those things listed aren’t enough, just head over to the additional script-page where you find tons of other script for other purposes. By default, Cacti comes with templates for the most common SNMP-setups.

So did we need to upgrade our server? Nope, as it turned out, we were doing fine.

By the way, due to security reasons you might want to disable the guest account in cacti.

Update 1: As the Cacti Developer Tony Roman points out, regardless if you use Cactid or not, you will still need a crontab job. The article previously stated that if you use cactid, you won’t need a crontab job, which was wrong.