This website demonstrates simple scripts that monitor servers by pinging them. They are designed to illustrate different strategies for monitoring, and can be used as a strating point for many situations.

Contents

• Monitoring
• check What we can check
• present How to present it
• os Operating Systems
• Screenshots and Downloads
• strategy1 Command Line Interface (CLI)
• strategy2 Website Report
• strategy3 CGI Report
• strategy4 Multi daemon

Monitoring

Check

The action that checks the service. The check may be to,

• ping the server
• connect to a port
• simulate an end-user event
• monitor performance

If it was a web server, a script could fetch a particular website and check it's MD5 and response time - tools like wget may assist. If a database server is to be tested, a database fetch could be attempted and the data checked - using Perl and CPAN libraries will help here. Using such specific tests is somtimes called "focused monitoring".

The use of focused monitoring tools cannot be emphasised enough - if support staff exist for a particular application, they should be using a focused monitoring tool. This is a tool that simulates what a client does to access the application, and can measure response times and sample content. Some companies have critical applications such that they hire a team of support staff for it. Waiting for the phone to ring is poor behaviour - the staff should have tools such that they know of problems before the customers do. If they don't have these tools, they should write them (or purchase them). It may well mean learning new languages.

Now, if we have gone to the effort to check a particular detail by writing our own script, it is helpful to have this script update a log whenever the check is performed. If a user were to say "the service was slow at 9am this morning", it is nice to have a log where we have recorded service response times from our end-user simulation script.


Presentation

This how we draw attention to problems. Some things to consider,

• web sites
• colour coding
• email alerts
• pager alerts
• audio alerts

Colour coding is very effective: red=bad, green=good. This is sometimes called "traffic lights", and will allow any staff member to understand your reports. Audio can be either keyboard beeps or recorded samples played through an audio card. Email or pager alerts should only be attempted if the code is "stateful" - a message is sent only when a change happens.


Operating Systems

The examples on this website are written for Unix or Linux, as they are ideal platforms to run monitoring tools from. This is because it is common for an install to have powerful scripting tools such as sh, ksh, bash, sed, awk; a powerful language with network libraries such as perl or python; tools to send emails such as sendmail, mail, mailx; and a webserver that may be needed to host the reports.

Other platforms such as Microsoft Windows could be used, however it may require installing extra software such as a perl distribution.

Screenshots and Downloads

The following demonstrates the operation of each of the programs with a link to download the code,

---

Strategy 1 - CLI

pinghosts is a simple command line program to ping hosts in /etc/hosts and colour the output. Variants could be written to ping an /etc/prodhosts file, or /etc/devhosts - whatever is suitable. Using files like this makes maintenance easy.


$ pinghosts Checking 127.0.0.1: 127.0.0.1 is alive Checking 192.168.1.1: 192.168.1.1 is alive Checking 192.168.1.2: no answer from 192.168.1.2 Checking 192.168.1.5: 192.168.1.5 is alive Checking 192.168.1.150: 192.168.1.150 is alive Checking 192.168.1.151: no answer from 192.168.1.151


In practise, this strategy is not as useful as it looks - it's only handy when you remember to run the command.



---

Strategy 2 - Website

getping.sh this pings servers and produces a colour coded html website which is served by a webserver. This could be scheduled to run via crontab to update every 5 minutes. Example website,


Ping began at Saturday January 25 22:33:20 EST 2003 venus,   venus is alive earth,   earth is alive mars,   mars is alive phobos,   no answer from phobos 192.168.1.1,   192.168.1.1 is alive Completed at Saturday January 25 22:33:26 EST 2003


In practise this strategy works quite well. This produces a website that can be viewed and understood by any staff member. The only problem is the time between updates where it may show stale data.



---

Strategy 3 - CGI

getping.cgi this pings servers and produces a colour coded html report. As a CGI it is triggered from a browser "on demand" to produce live data. Example website,


Ping began at Saturday January 25 22:34:30 EST 2003 venus,   venus is alive earth,   earth is alive mars,   no answer from mars phobos,   phobos is alive 192.168.1.1,   192.168.1.1 is alive Completed at Saturday January 25 22:34:36 EST 2003


In practise this produces the best results. However if it is used by many staff simultaneously it may generate a strain on the network. It may be best accessed from a protected area of the web server, for IT support staff only.



---

Strategy 4 - Multi (Email, Syslog, CLI and Web)

watchping is a watchdog program to ping servers and take action if they go down. It is designed to run as a daemon so that it can be stateful - eg, only email when something changes not every time a check is made. The four actions are to send email alerts, send syslog alerts, log everything, and generate a website. By default it uses email and syslog.

Example website,


WatchPing Report, Friday January 3 03:07:13 EST 2003 mars is alive no answer from phobos localhost is alive 127.0.0.1 is alive no answer from 192.168.1.5

Example running 1,

Here, watchping is run in verbose mode for the hosts mars and phobos. Phobos is down, so it sends a syslog message, emails root a message, and prints messages to the screen (verbose),


# ./watchping -v mars phobos Running WatchPing... Sleep Interval: 60 secs Email address: root Syslog priority: user.err Checking Hosts: mars phobos ----- Friday January 3 02:34:37 EST 2003 mars is alive no answer from phobos ----- Friday January 3 02:35:42 EST 2003 mars is alive no answer from phobos # tail -1 /var/adm/messages Jan 3 02:34:42 mars watchping: [ID 702911 user.alert] Hosts Down: phobos # mail From root@mars.dev.com Fri Jan 3 02:35:47 2003 Date: Fri, 3 Jan 2003 02:35:47 +1100 (EST) From: Root Message-Id: <200301231535.h0NFzLVF029404@mars.dev.com> Subject: WatchPing Alert: phobos Content-Length: 159 WatchPing Alert The following hosts failed to ping, phobos Output from all pings, Friday January 3 02:35:42 EST 2003 mars is alive no answer from phobos ?

Example running 2,

Watchping can be used as a background process on startup, configured to check against custom lists of hosts, email different address, etc. This example demonstrates a combination of actions,


watchping -e sysadmin@mars -w /var/http/prod.html -i /etc/prod.txt & watchping -e dbadmin@venus -w /var/http/db.html -i /etc/db.txt &


Both of the above run in the background, check every 60 seconds, and send alerts to syslog. They will also email alerts to the address after "-e", they read host lists from the text files in /etc after the "-i" (here one may list production server names, the other database server names), and they even update colour coded websites after the "-w" (red is bad, green is good, blue is unknown).
In practise, complex strategies like the above depend on your environment. There may be political, departmental or policy reasons for email, websites or logs to be used in a certain way.

If I've passed on some ideas for monitoring then the goal of this website has been a success. Good luck!