Using SNMP and RRDTool on FreeBSD

Statistics make the world go round. They help us to see the big picture, and stop us becoming headless chickens.

This article applies equally to FreeBSD 4.x and 5.x

The method of collecting statistics has evolved over thousands of years, but the basic process is still very much the same: generate numbers based on work done, and then make it presentable. How exactly you make it presentable is very much up to you. Some people would choose to tabularize it, others to graph it. And then there are some who like to look at data in it's raw form. How you choose to present the data should depend on what is it you are trying to measure. In this paper we will be generating various system related numbers and then using rrdtool to graph them. The whole process should about half an hour and not more than an hour, after which time it is mostly self sustaining.

The software of choice

When we are generating our statistical data, we certainly are not limited in choice by which applications we can use. Indeed we don't even have to use SNMP. Most UNIX based operating systems come with a variety of tools we could use, ranging from home made scripts to almost any program which tells you something about your system.

SNMP does offer us a number of advantages which are otherwise hard to achieve. Most importantly I believe, it offers a central location from which we can take numbers for a range of data sources. I have in the past used SNMP to retrieve data on various network counters and system load, which is a reasonably typical use for snmp. This is also what we will be going over here.

After we have gathered our data, we to make it presentable. Raw numbers are sometimes fine but when you need to express that data to your management or even track it over the long term for yourself, then you need to store and analyze it. MRTG, RRDTool and various other packages are available to help you do this. MRTG (Multi Router Traffic Grapher) was originally designed, as the name implies, to graph the traffic data from multiple routers. This was then adapted by many admins to graph data from various other sources. RRDTool was born of this effort, as a reimplementation of MRTG's graphing and logging features. RRD is an abbreviation of Round Robin Database, which describes the manner in which the program stores it's data. For those interested in RRDTool outside the scope of this document, I would recommend a visit to the RRDTool website: http://www.rrdtool.org/

Installing the software

FreeBSD (as well as NetBSD and OpenBSD) have arguably one of the best package management systems available today. The simplicity and flexibility are something a few other package management systems could do well to learn from. We will take full advantage of this when installing our software to save us time and effort:

# cd /usr/ports/net-mgmt/net-snmp && make install clean
# cd /usr/ports/databases/rrdtool && make install clean

These two commands should install Net-SNMP, and RRDTool. If you have problems at this stage, seek your local BSD guru.

Configuring snmpd

The net-snmp package comes with the snmpconf utility which should help you configure snmpd. We will walk through the steps for it, as it can be somewhat harrowing for those new to snmp:

```
# snmpconf -i

          I can create the following types of configuration files for you.
          Select the file type you wish to create:
          (you can create more than one as you run this program)

             1:  snmp.conf
             2:  snmptrapd.conf
             3:  snmpd.conf
```
snmp.conf (snmp.conf(5)) dictates how Net-SNMP applications should operate, snmptrapd.conf (snmptrapd.conf(5)) configures the snmptrapd daemon which is used for on going monitoring via SNMP, and finally snmpd.conf (snmpd.conf(5)) defines how the snmp daemon which we will be using, is configured. Choose option '3'.
The main menu:
```
             1:  System Information Setup
             2:  Access Control Setup
             3:  Trap Destinations
             4:  Monitor Various Aspects of the Running Host
             5:  Extending the Agent
             6:  Agent Operating Mode
```
Choose option '1', and at the 'System Information Setup' menu, choose '1' again. You will be asked to enter the location of your system. This can be the country, locality, your street name, site name, or any other data you would use to describe the location of the system. After you enter this, you will be returned to the 'System Information Setup' menu.
Choose option '2' and enter the contact information for the administrator of the machine.
Finally, when you choose option '3', you will be asked a number of Yes/No questions to determine what the proper value for the sysServices object. This is used to determine what services your machine offers. Reply with a '1' for yes, or '0' for no.
When you are done with the 'System Information Setup' menu, type 'finished' to return to the main menu.
From the main menu, choose option '2' for 'Access Control Setup'
```
             1:  a SNMPv3 read-write user
             2:  a SNMPv3 read-only user
             3:  a SNMPv1/SNMPv2c read-only access community name
             4:  a SNMPv1/SNMPv2c read-write access community name
```
For the purpose of simplicity, we will set up a read-only SNMPv1 server which listens only on localhost. SNMPv2 and v3 provide some added security in the form of usernames and passwords, and for anyone wishing to have their snmpd listen on the network, I would very seriously recommend they look in to using these.
To continue, choose option '3'. You will be asked to enter a community name for read-only access. This can be any one-word string you wish, and is simply used as a weak method of authentication to restrict access to the service.
Then choose 'localhost' as the hostname from which to accept that community string, and press RETURN for no-restrictions on what the user of that community string can read.
Next we will confirm where the daemon should be listening. From the main menu, choose option '6' for 'Agent Operating Mode'
```
             1:  Should the agent operate as a master agent or not.
             2:  The system user that the agent runs as.
             3:  The system group that the agent runs as.
             4:  The IP address and port number that the agent will listen on.
        
```
Choose option '4', and enter 127.0.0.1 as the address at which snmpd will listen. Return to the main menu, and choose 'finished' again. You will be shown to the original menu asking which file you would like to edit. choose to 'quit'.
Finally, edit the /etc/rc.conf startup file, and at the end of the file put a new line as:
snmpd_enable="YES"
You can now start snmpd with /usr/local/etc/rc.d/snmpd.sh start. It will also start with your system when you reboot.

Configuring RRDTool

If you thought configuring snmpd was easy, you'll appreciate RRDTool.
As mentioned about, RRD is an abbreviation of Round Robin Database, which describes the method of data storage for the RRDTool application. You start by defining and creating a simple database which will hold your data. Your database can only hold a fixed amount of data. Once this database has been "filled" with new data, the oldest data is overwritten. In this way, the data entry goes in a round robin, and ultimately there is no "start" or "end" to the database. It is just one loop.

An rrd database is created with the 'rrdtool create' command. I find it helpful to store all of my rrd related user files in a single directory, /usr/local/rrd.

# mkdir /usr/local/rrd && cd /usr/local/rrd
# rrdtool create bandwidth.rrd --start N DS:in:COUNTER:600:U:U DS:out:COUNTER:600:U:U RRA:AVERAGE:0.5:1:432

This looks scary, but it really isn't. Here is how it breaks down:

rrdtool create bandwidth.rrd
This tells rrdtool you want to make a new database called bandwidth.rrd
--start N
For graph plotting purposes and also for data consistency, rrdtool needs to know the time of each data entry and also the time when the database's first entry is to be accepted. rrdtool will not accept any data before that time. 'N' tells rrdtool that the starting time is now.
DS:in:COUNTER:600:U:U
This option tells rrdtool we want to define a data source for which data will be saved. It breaks down like this:
- DS
  instructs that we are defining a data source.
- in
  is the name of the data source, which in this case is describing that we want to store data for the amount of network traffic inbound.
- COUNTER
  Everybody has different requirements for where their data will come from. Some counters reset themselves to zero after being read, some go up and down, and some only ever go up! rrdtool is able to handle this by letting you define what type of source is being read. Here we choose COUNTER; COUNTER sources only increment. To see the other types of data sources, read the rrdcreate(1) man page.
- 600
  rrdtool also needs to know how often you will be entering data. It will use this to work out intermediate values in between checks, as well as various other tasks. This is called the heartbeat. If no data is received for more than the length of the heartbeat, the data for that interval will get marked *UNKNOWN*
- U:U
  We can optionally define minimal and maximal values which are valid. This is used to prevent one erroneously large or small value from compromising the integrity of the data you store. While this can be useful in some other situations, for us it is not. We define 'U' for each of these to leave them unknown
RRA:AVERAGE:0.5:1:432
In the final part of our create command, we define our Round Robin Archive. Incoming data is stored in the RRA part of our round robin database, and how it is stored depends on a number of options we supply for the RRA.
- RRA
  Define the start of the RRA option.
- AVERAGE
  As part of the RRA, we must also define a consolidation function. The archive does not keep all of the data it is sent. Rather, it applies the consolidation function to the inbound data. For example, if you are gathering data every minute, but your RRA stores data from every 5th data entry, then the consolidation function is applied to work out what is actually stored. If the AVERAGE function is applied, then in this example the average for the last 5 data entries is stored. The data which gets stored is called the consolidated data point.
- 0.5
  The most confusing part of rrdtool for me, is setting up the RRA. This value is the xff, or xfiles factor. I don't understand it, and I don't pretend to. However, every example I have ever seen tells me to use 0.5, so I do too. The xff defines how much data in a consolidation function can be made up of unknown data. I take this to mean that with a value of 0.5, up to half of my data can be *UNKNOWN*.
- 1
  As we mentioned for the consolidation function (AVERAGE), we define how many of our data entries (primary data points, or PDP's) are used to create an average consolidated data point. If we choose '1', all of our data points will be stored, and none will be averaged.
- 432
  Finally we want to define how large our database is. How many data points do we want to store? A day's? A month's? A year's? Previously we specified that we would enter data once every 600 seconds. If we keep 432 rows of data, that is 259200 seconds, or 72 hours.
You can also specify multiple RRA's to record your data. The following RRA will store 18 days of data, averaged at every 6th data entry:
RRA:AVERAGE:0.5:6:432

Adding data to your Round Robin Database

Now that we have everything configured, we want to start adding data into our newly created database. Before we can do that, we need to look at our snmp tree and find out where the data we want is. The command for this would be:

# snmpwalk -v 1 -c <your community string> localhost

The first string you are looking for is IF-MIB::ifDescr.? where the ? is a number greater than 0. This string will tell you the number of the interface you want to monitor. For example, the string I needed is: IF-MIB::ifDescr.1 = STRING: sis0, which tells me to use interface '1'.

The next command will run snmpget to fetch the byte counter values for the inbound and outbound statistics:

# snmpget -v 1 -c <your community string> -Oqv localhost IF-MIB::ifInOctets.1 IF-MIB::ifOutOctets.1

Replace the '1' in 'IF-MIB::ifInOctets.1' and IF-MIB::ifOutOctets.1', with the number of the interface you are monitoring. If you omit the '-Oqv' you get a more verbose result.
The next step is to get this data into rrdtool. The RRDTool application accepts data from the command line in the format: rrdupdate <rrd file> timestamp:value1:value2:...
We have seen in the example for creating the database itself, that we can use the letter 'N' to represent the current time. The values must be specified in the order in which they were created in the database. In our example the DS for inbound traffic came before the DS for outbound traffic, so we will specify them in that order when we update the database.

Some readers may have spotted a problem here. The output from our snmpget command is in a different format than we need. I'm sure that as you read this you're trying to think of different solutions to the problem. Indeed I can think of at least 5 different ways to get the data into the format we want.The simplest answer may be a command like this:

rrdupdate /usr/local/rrd/bandwidth.rrd N:\
      `/usr/local/bin/snmpget -v 1 -c <your community string> -Oqv localhost IF-MIB::ifInOctets.1`:\
      `/usr/local/bin/snmpget -v 1 -c <your community string> -Oqv localhost IF-MIB::ifOutOctets.1`

When the database was created, we specified an update interval of no more than 600 seconds (10 minutes). FreeBSD's cron(8) allows a user to say with ease that a command should be run every 5 minutes, using a line like this:
0-55/5 * * * * /usr/local/bin/rrdupdate /usr/local/rrd/bandwidth.rrd N:`/usr/local/bin/snmpget -v 1 -c <your community string> -Oqv localhost IF-MIB::ifInOctets.1`:`/usr/local/bin/snmpget -v 1 -c <your community string> -Oqv localhost IF-MIB::ifOutOctets.1` You should make sure when you add this to your crontab(1), that it is all on one line.

Getting data from your Round Robin Database

We have now installed our software, created our database and started adding data to it. RRDTool like MRTG, has the ability to generate some rather nice graphs in GIF, PNG or GD formats. You should be aware that there are licensing restrictions around creating GIF format files. As such most users will want to create PNG format graphics as they are the most portable (PNG is an abbreviation of Portable Network Graphics).

While the rrdtool graph command is relatively easy to understand, it is also very long winded. For this reason it is often easiest to put the command in a simple shell script. So, open your favourite text editor and drop this in:

#!/bin/sh
/usr/local/bin/rrdtool graph /usr/local/rrd/bandwidth.png -a PNG -h 125 -s -129600 -v "Data Throughput" \
    'DEF:in=/usr/local/rrd/bandwidth.rrd:in:AVERAGE' \
    'DEF:out=/usr/local/rrd/bandwidth.rrd:out:AVERAGE' \
    'CDEF:kbin=in,1024,/' \
    'CDEF:kbout=out,1024,/' \
    'AREA:in#00FF00:Bandwidth In'                               'LINE1:out#0000FF:Bandwidth Out\j' \
    'GPRINT:kbin:LAST:Last Bandwidth In\:    %3.2lf KBps'       'GPRINT:kbout:LAST:Last Bandwidth Out\:   %3.2lf KBps\j' \
    'GPRINT:kbin:AVERAGE:Average Bandwidth In\: %3.2lf KBps'    'GPRINT:kbout:AVERAGE:Average Bandwidth Out\:%3.2lf KBps\j'

The options break down like this:

DEF:
The DEF line defines what data will be extracted from the round robin database. We are extracting from the bandwidth.rrd file, from the 'in' and 'out' RRA's, and we want to graph the AVERAGE RRA's values.
CDEF:
RRDTool allows us to play with the data we have extracted before we graph it, by performing various mathematical functions against it. Here were are creating new values called 'kbin' and 'kbout' by taking the 'in' and 'out' values, taking the number 1024, and applying the division function. This translates the byte values from our counters into kilobytes, which are a little easier to read. When we draw the graph we will use the byte values, as rrdtool can scale and label our y-axis to the appropriate byte, kilobyte or megabytes. We can also print a simple key under the graph with the current data, and this is where we use the CDEF values.
AREA and LINE
Our graph can contain areas and lines. Typically inbound traffic is shown as an AREA, and outbound traffic as a LINE. This isn't a requirement, just a tradition. We specify what we want to graph, the colour we want to draw it in, and what label will be associated with it. When the graph is created, a legend is printed directly beneath it showing the viewer which label is associated with each colour. The '\j' at the end of the line dictates that this legend should be printed justified (spread across the bottom of the graph) and not crunched in the center or to one side.
GPRINT
The GPRINT option creates the afore mentioned key at the bottom of the graph, under the legend. Here we have chosen to print the kbin value from the CDEF along with the words "Last Bandwidth In" to two decimal places. The same is done for the Last Bandwidth Out. Notice the word LAST in capitals? This informs rrdtool, that when calculating the kbin and kbout values, it should use the last value in the RRA, not the AVERAGE. The possible options here are AVERAGE, MIN, MAX and LAST. As you can see, multiple GPRINT statements can be defined, so creating a key for the minimum and maximum values in the RRA should be a simple task.

Save your script as rrd-graph.sh, and remember to change it's permissions to allow executing:
# chmod 755 /usr/local/bin/rrd-graph.sh
Wait 30 minutes to allow several updates to your database and then run your graphing script:
# /usr/local/bin/rrd-graph.sh
You should be left with a PNG graphic of your data. In this fashion you can graph almost anything the snmp daemon can tell you about your system. Here are a few other values you can use, try creating your own round robin databases for them, and then plotting their graphs:

UCD-SNMP-MIB::laLoadInt.1, UCD-SNMP-MIB::laLoadInt.2 and UCD-SNMP-MIB::laLoadInt.3
The 1, 5 and 15 minute load averages for your system, multiplied by 100 to make them integers
UCD-SNMP-MIB::memAvailReal.0
The amount of free physical memory in your system
HOST-RESOURCES-MIB::hrSystemProcesses.0
The number of processes currently running on this system

If you get bored with these and want to see which over values snmpd can give you, look at the files in /usr/local/share/snmp/mibs. The files are named MIB.txt, and contain entries like this:

HOST-RESOURCES-MIB.txt:
hrSystemProcesses OBJECT-TYPE
    SYNTAX     Gauge32
    MAX-ACCESS read-only
    STATUS     current
    DESCRIPTION
        "The number of process contexts currently loaded or
        running on this system."
    ::= { hrSystem 6 }

So to monitor this value, you would fetch HOST-RESOURCES-MIB::hrSystemProcesses.0.
Notice that I added an extra '.0' at the end. snmpd can monitor multiple values in the same category (look at the 1, 2 and 3 in the system load averages example above). If you;re unsure what number should be appended, start at zero and work your way up.

Try to find the MIB that will let you count the number of apache processes, or to check the accounting for IPFW.

What is WraithNet?

WraithNet is a personally run network for free web hosting for anyone publishing documents or data relating to the open-source community.
If you would like a sub-domain for your mail and web pages, send an email to:
webmaster@silverwraith.com

References and Credits:

This is a list of references I found when writing this paper. Most of them contain much more detail on their particular subject than I have provided here. Hopefully they will be as useful to you as they were to me.

Credits go to Nina F. for suggesting the topic of this paper.

If you find a problem with this website, please email:
webmaster@silverwraith.com