Using InfluxDB + Grafana to Display Network Statistics
I loathe MRTG graphs. They were cool in 2000, but now they’re showing their age. We have much better visualisation tools available, and we don’t need to be so aggressive with aggregating old data. I’ve been working with InfluxDB + Grafana recently. Much cooler, much more flexible. Here’s a walk-through on setting up InfluxDB + Grafana, collecting network throughput data, and displaying it.
influxsnmpis no longer updated, and the instructions below won’t work properly. Check out this new blog on how to use Telegraf with InfluxDB and Grafana instead.
Background - InfluxDB + Grafana
There’s three parts to this:
- Grafana: This is our main UI. Grafana is a ”…graph and dashboard builder for visualizing time series metrics.” It makes it easy to create dashboards for displaying time-series data. It works with several different data sources such as Graphite, Elasticsearch, InfluxDB, and OpenTSDB.
- InfluxDB: This is where we store the data that Grafana displays. InfluxDB is “…an open-source distributed time series database with no external dependencies.” It’s a relatively new project, and is not quite at 1.0 yet, but it shows a lot of promise. It can be used in place of Graphite. It is very flexible, and can store events as well as time series data.
- Influxsnmp: We need to get data from the network into InfluxDB. There are a few options for doing this, but none of them are particularly good right now. Influxsnmp is a simple utility for collecting data via SNMP and storing it in InfluxDB.
You might ask why I’m using separate collection, persistent store and presentation layers, rather than using one system that does it all. The advantage of this model is that is much more flexible - I can easily add collectors to get data from different sources, or I can use other data stores that I already have. If I later decide I don’t like Grafana, I can query InfluxDB from something else.
All steps are taken on an Ubuntu 14.04 system. Steps may vary for other systems. Note also that I’m using the nightly builds for InfluxDB + Grafana, so expect to see the outputs change slightly in future.
Download, install and start InfluxDB:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 lhill@grafana:~$ curl -O https://s3.amazonaws.com/influxdb/influxdb_nightly_amd64.deb % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14.1M 100 14.1M 0 0 433k 0 0:00:33 0:00:33 --:--:-- 416k lhill@grafana:~$ sudo dpkg -i influxdb_nightly_amd64.deb [sudo] password for lhill: Selecting previously unselected package influxdb. (Reading database ... 95702 files and directories currently installed.) Preparing to unpack influxdb_nightly_amd64.deb ... Unpacking influxdb (0.9.5-nightly-9633410) ... Setting up influxdb (0.9.5-nightly-9633410) ... Removing any system startup links for /etc/init.d/influxdb ... Adding system startup for /etc/init.d/influxdb ... /etc/rc0.d/K20influxdb -> ../init.d/influxdb /etc/rc1.d/K20influxdb -> ../init.d/influxdb /etc/rc6.d/K20influxdb -> ../init.d/influxdb /etc/rc2.d/S20influxdb -> ../init.d/influxdb /etc/rc3.d/S20influxdb -> ../init.d/influxdb /etc/rc4.d/S20influxdb -> ../init.d/influxdb /etc/rc5.d/S20influxdb -> ../init.d/influxdb lhill@grafana:~$ sudo service influxdb start Starting the process influxdb [ OK ] influxdb process was started [ OK ] lhill@grafana:~$
Now use the InfluxDB CLI client to create two databases. We’re going to use one for SNMP data, and another for storing events. Note that we don’t need to declare schemas.
1 2 3 4 5 6 7 8 lhill@grafana:~$ /opt/influxdb/influx Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring. Connected to http://localhost:8086 version 0.9.5-nightly-9633410 InfluxDB shell 0.9.5-nightly-9633410 > create database snmp > create database events > exit lhill@grafana:~$
We now have InfluxDB running. You can connect to
http://localhost:8083/ for a simple Web UI to InfluxDB, or query it with the CLI client.
Influxsnmp is written in Go. Today it is only distributed as source, so we need to build it. The build process includes some components that require Go >= v1.3. The default repositories for Ubuntu 14.04 have Go v1.2.1. So for this setup we’re going to download the latest Golang binary package, and use that.
First install git, then get Go 1.5.1, and setup our environment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 lhill@grafana:~$ sudo apt-get install git Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: git-man Suggested packages: git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-arch git-bzr git-cvs git-mediawiki git-svn The following NEW packages will be installed: git git-man 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 3,325 kB of archives. After this operation, 21.5 MB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://nz.archive.ubuntu.com/ubuntu/ trusty-updates/main git-man all 1:1.9.1-1ubuntu0.1 [698 kB] Get:2 http://nz.archive.ubuntu.com/ubuntu/ trusty-updates/main git amd64 1:1.9.1-1ubuntu0.1 [2,627 kB] Fetched 3,325 kB in 2s (1,133 kB/s) Selecting previously unselected package git-man. (Reading database ... 95719 files and directories currently installed.) Preparing to unpack .../git-man_1%3a1.9.1-1ubuntu0.1_all.deb ... Unpacking git-man (1:1.9.1-1ubuntu0.1) ... Selecting previously unselected package git. Preparing to unpack .../git_1%3a1.9.1-1ubuntu0.1_amd64.deb ... Unpacking git (1:1.9.1-1ubuntu0.1) ... Processing triggers for man-db (184.108.40.206-1ubuntu1) ... Setting up git-man (1:1.9.1-1ubuntu0.1) ... Setting up git (1:1.9.1-1ubuntu0.1) ... lhill@grafana:~$ curl -O https://storage.googleapis.com/golang/go1.5.1.linux-amd64.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 74.2M 100 74.2M 0 0 1010k 0 0:01:15 0:01:15 --:--:-- 1126k lhill@grafana:~$ tar zxf go1.5.1.linux-amd64.tar.gz lhill@grafana:~$ echo 'export GOROOT=$HOME/go' >> .profile lhill@grafana:~$ echo 'export PATH=$PATH:$GOROOT/bin' >> .profile lhill@grafana:~$ echo 'export GOPATH=$HOME/work' >> .profile lhill@grafana:~$ . .profile lhill@grafana:~$
Now download the source for influxsnmp and build it:
1 2 3 4 5 6 lhill@grafana:~$ go get github.com/paulstuart/influxsnmp lhill@grafana:~$ go install github.com/paulstuart/influxsnmp lhill@grafana:~$ ls $GOPATH/bin/ influxsnmp lhill@grafana:~$ cp $GOPATH/src/github.com/paulstuart/influxsnmp/oids.txt $GOPATH/bin/ lhill@grafana:~$ mkdir $GOPATH/etc
The packaging for influxsnmp isn’t quite right yet, so we have to manually copy over the oids.txt file. There are sample config.gcfg & ports.txt files that ship with influxsnmp. They’re easy to understand. Here’s what mine look like, for polling a few interfaces on my SRX. Note the location of ports.txt - it needs to be in the same directory as the binary file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 lhill@grafana:~$ cat $GOPATH/etc/config.gcfg ; multiple snmp devices can be specified ; their config name must match a mib config name ; or an alternate config name can be used ; or there must be a wildcard mib configured -- named '*' [snmp "srx110"] host = srx community = Sup3rS3cr3t port = 161 timeout = 20 retries = 5 repeat = 0 freq = 30 ;debug = false ; if port file is ommited then all columns will be retrieved portfile = ports.txt ; this is a wildcard -- becomes default ; if a 'snmp' section name is not otherwise specified [mibs "*"] name = ifXEntry scalers = false column = ifHCInOctets column = ifHCInUcastPkts column = ifHCOutOctets column = ifHCOutUcastPkts column = ifInErrors column = ifInDiscards column = ifOutErrors column = ifOutDiscards [influx "*"] host = localhost port = 8086 db = snmp ;user = username ;password = password ; web status monitor - set port to 0 to disable [http] port = 9501 lhill@grafana:~$ cat $GOPATH/bin/ports.txt # snmp_column label_for_metrics fe-0/0/4 NetBeez fe-0/0/5 AppleTV fe-0/0/6 Wireless fe-0/0/7 TiVo at-1/0/0 ADSL
Finally we’ll start the poller:
1 2 3 4 5 6 lhill@grafana:~$ nohup $GOPATH/bin/influxsnmp -config $GOPATH/etc/config.gcfg -logs $HOME/log > $HOME/log/nohup.out 2>&1 & lhill@grafana:~$ cat log/nohup.out nohup: ignoring input Web interface: http://192.168.166.164:9501 lhill@grafana:~$
(Yes, it really should be properly installed as a service, with proper config file locations, logging, etc.)
There’s a simple http interface listening on port 9501. That has some basic stats on influxsnmp operations. We can also take a look at InfluxDB to check that we’re receiving data:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 > select value from ifHCInOctets where column='ADSL' name: ifHCInOctets ------------------ time value 1445821685851115439 52912945362 1445821692356103691 52912957099 1445821722367440151 52913306572 1445821752356421192 52914402662 1445821782357405741 52914782128 1445821812356691368 52914913911 1445821842357547023 52915284111 1445821872357314936 52915831952 1445821902357408200 52915932362 1445821932357485730 52915948310 1445821962356608706 52916072093 >
So far so good - we’re collecting data, storing it, and we can query it with InfluxDB. Now we need to create visualisations.
First download, install & start Grafana. Note that I’m using a nightly build again.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 lhill@grafana:~$ curl -O https://grafanarel.s3.amazonaws.com/builds/grafana_latest_amd64.deb % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 22.0M 100 22.0M 0 0 568k 0 0:00:39 0:00:39 --:--:-- 608k lhill@grafana:~$ sudo dpkg -i grafana_latest_amd64.deb Selecting previously unselected package grafana. (Reading database ... 96458 files and directories currently installed.) Preparing to unpack grafana_latest_amd64.deb ... Unpacking grafana (2.5.0-pre1) ... Setting up grafana (2.5.0-pre1) ... Adding system user `grafana' (UID 105) ... Adding new user `grafana' (UID 105) with group `grafana' ... Not creating home directory `/usr/share/grafana'. ### NOT starting grafana-server by default on bootup, please execute sudo update-rc.d grafana-server defaults 95 10 ### In order to start grafana-server, execute sudo service grafana-server start Processing triggers for ureadahead (0.100.0-16) ... ureadahead will be reprofiled on next reboot lhill@grafana:~$ sudo service grafana-server start * Starting Grafana Server [ OK ] lhill@grafana:~$
Grafana is now listening on port 3000. Open a web browser, and login using the default admin/admin. Click on Data Sources on the left:
Click Add new and fill in these details:
- Name: snmp
- Default: selected
- Type: InfluxDB 0.9.x
- Http settings: - Url
- InfluxDB Details: - set Database to ‘snmp’. We’re not using a username & password, but you need to put something in there to be able to save this. So use admin/admin
After you’ve clicked Add, you’ll be able to click Test Connection - try it, and make sure that it’s OK.
Repeat the above process for a new Data Source, with two differences - set the Database as “events” and leave “Default” unchecked. We’ll use this later for annotations.
Now we’re ready to go with creating dashboards.
Visualising the Data
Grafana lets you have multiple dashboards. Each dashboard is a set of panels, where a panel could be a graph, a single metric, or some text. You could have a dashboard with 10 separate graphs, or maybe just one huge number. We want to set up a simple dashboard that shows us a graph of the current traffic on my ADSL link.
Click on Dashboards on the left, then the drop-down arrow beside Home. Click New at the bottom:
On the new blank dashboard, click the small bar at the top left. That will expand out. Choose Add Panel -> Graph
Now you’ll get a graph with test data. Click on the title, where it says “no title (click here).” On the pop-up box, click Edit.
Click on the General tab, and set the title to “ADSL Throughput.” Click on Metrics. Grafana’s query editor works well for regular measurements, but it doesn’t currently work for derivative functions. We’ll use the editor to build up our query, and then switch to raw mode to finish off.
Next to FROM, click “select measurement” - this will change to a drop-down that should list the different measurements we’re collecting. We want “ifHCInOctets.” On the WHERE _line, click +, select “column” = “ADSL” You’ll need to choose one of your own interfaces, as defined in ports.txt. On the _ALIAS BY line, enter “Input.”
Click the “+ Query” button, and add a new query with the same settings as above, except select “ifHCOutOctets.”
Now we need to edit the queries in ‘raw’ mode. Your query screen should look like this:
Click the menu button in the top left, and click “Switch editor mode.” We need to add the
derivative() function, because we’re collecting a counter, not a rate. We also need to multiply the result by 8, to get bps. Your queries should look like this:
1 SELECT 8 * derivative(mean("value"),1s) AS "value" FROM "ifHCInOctets" WHERE "column" = 'ADSL' AND $timeFilter GROUP BY time($interval) fill(none)
One more thing to do - click on Axes & Grid and click “short” next to Unit. From the drop-down, go data rate -> bits/sec. If you like, you can also enable Legend values: Min, Max, Avg, Current.
Click Back to dashboard. Click on the Gear icon near the top centre of the screen, and choose Settings. Give the dashboard a title, and click the Save dashboard icon.
You should now have a screen that looks something like this:
Play around with the time picker in the top-left of the screen, and try selecting & zooming into a time window. It will get more interesting as you collect more data.
That graph is OK, but what about having a graph per interface? Or having a drop-down list of interfaces, so we can choose which ones get displayed? This is where the Grafana Template functionality is useful. Click on the Dashboard Gear icon, and click Templating. Click “+New”
Set Name: _Interface, leave it at _Type: query, Data Source: snmp. Set the Query _to
SHOW TAG VALUES WITH KEY = "column" and enable Multi-value selection. Click _Add. Close the Templating box.
Edit the original graph. Change each query to that it looks for the variable “$Interface” instead of ‘ADSL.’ On the General tab, change the title to “$Interface Throughput.” Set Repeat Panel to “Interface.” Save the Dashboard.
There will be a drop-down list of Interfaces at the top of the Dashboard. Choose a different interface, and watch the graph update. Select two interfaces at once, and see how you get two graphs:
One of the nice things about InfluxDB + Grafana is that it is easy to add events to a graph. These are called annotations. They could be anything - e.g. a link state change, or maybe a production code deployment notification. Here I’ve stored my events in InfluxDB, but you could pull them from somewhere else - e.g. Elasticsearch. Ideal if you have your syslogs going into an ELK stack.
For this example I want to add annotations when I start & stop watching a TV show on Netflix. This should show up in my traffic stats.
Go to Dashboard settings -> Annotations. Click “+New” and call it “Netflix.” Make sure the Datasource is set to “events.” The query should be:
SELECT * FROM "alert" WHERE $timeFilter and for the Column mappings, set Title to “show”, and Text to “text.” Close the Annotations box.
Go to the CLI, and manually stuff in some events:
1 2 3 4 5 6 7 8 9 10 11 12 13 lhill@grafana:~$ curl -i -XPOST 'http://localhost:8086/write?db=events' --data-binary 'alert,host=srx,show=Spartacus text="Started watching Netflix" 1445475780000000000' HTTP/1.1 204 No Content Request-Id: 9d7642cc-7ba3-11e5-81d9-000000000000 X-Influxdb-Version: 0.9.5-nightly-9633410 Date: Mon, 26 Oct 2015 05:37:15 GMT lhill@grafana:~$ curl -i -XPOST 'http://localhost:8086/write?db=events' --data-binary 'alert,host=srx,show=Spartacus text="Finished watching Netflix" 1445477318374301617' HTTP/1.1 204 No Content Request-Id: b6419d5f-7ba3-11e5-81de-000000000000 X-Influxdb-Version: 0.9.5-nightly-9633410 Date: Mon, 26 Oct 2015 05:37:57 GMT lhill@grafana:~$
Now we get annotations on our graph. Hover over them, and the tool-tip displays some more info:
One small niggle is that you can’t disable annotations on a per-graph basis, only on a per-dashboard basis. Still, it’s very handy.
Obviously there’s a lot that could be improved upon here:
- Sort out the SNMP data capture - either package up influxsnmp properly, or (preferred answer): write a plugin for Telegraf.
- Write Ansible + Vagrant config to package up the above
- Even better: Do it with Docker, based upon Brent’s work.
- Add more network data sources - e.g. system-level stats, or NOS data collected via API.
The point is that it’s a flexible framework, and it doesn’t need to just be network stats. It gets much more interesting once you start adding in other data sources.