Stack Setup and Config Guide: Graphite + Grafana + StatsD
This lengthy article will guide you through installation, configuration and getting started with the amazing metric stack that is composed of Graphite, Grafana and StatsD.
Graphite is still king when it comes to time series databases due to its simple data model, ingestion with integrated aggregation and rollups, amazing query features and speed. No other time series database has yet to match Graphite’s query flexibility and analytics potential.
Graphite has a reputation for being tricky to install and scale. This guide aims to show that is not really the case, or, at least, that it is a lot better than you expect.
This guides does not only aim to be only be an install guide but to also teach you of the mechanics of metric collection, aggregation and querying. How Graphite stores and aggregates data is very important to understand in order to not get mislead by graphs.
Installation - Ubuntu
To begin with we are going to install the 3 main components that define our metric stack. Later in the guide we will install StatsD, but that is optional.
- Carbon is the graphite ingestion daemon responsible for receiving metrics and storing them.
- Graphite-api is light weight version of graphite-web with only the HTTP API and is responsible for executing metric queries.
- Grafana as the frontend to visualize metrics and the tool to help you build metric queries that will make the most out of your collected metrics.
Graphite and Carbon are written in python, so we will start by installing python packages.
apt-get install \ git \ build-essential \ libffi-dev libcairo2-dev \ python-django \ python-django-tagging \ python-simplejson \ python-memcache \ python-ldap \ python-cairo \ python-twisted \ python-pysqlite2 \ python-support \ python-dev \ python-pip
Next we will clone carbon and whisper and install these components. Whisper is just a lib used by carbon to write metrics to disk.
cd /usr/local/src git clone https://github.com/graphite-project/carbon.git git clone https://github.com/graphite-project/whisper.git cd whisper && python setup.py install && cd .. cd carbon && python setup.py install && cd ..
Copy example carbon config:
cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
Edit the config file
/opt/graphite/conf/carbon.conf, find line
change this setting to
Create a new file at
/opt/graphite/conf/storage-schemas.conf with the following content:
[carbon] pattern = ^carbon\..* retentions = 1m:30d,10m:1y,1h:5y [default] pattern = .* retentions = 10s:1d,1m:7d,10m:1y
This config specifies the resolution of metrics and the retention periods. For example for all metrics beginning with the word
carbon receive metrics every minute and store for 30 days, then
roll them up into 10 minute buckets and store those for 1 year, then roll those up into 1 hour buckets and store those for 5 years. For all other metrics
the default rule will be applied with other retention periods.
This configuration is very important, as the first retention period must match the rate at which you send metrics. The default rule has 10 seconds as its first resolution so when configuring StatsD we should configure it to send metrics every 10 seconds.
If you send values more frequently than the highest resolution, for example if you send data every second but the storage schema rules defines the highest resolution to be 10 seconds, then the values you send will just overwrite each other and the last value sent during every 10 second period will be saved. StatsD can work around this problem.
Copy the default config and open it in an editor.
cp /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf
[min] pattern = \.min$ xFilesFactor = 0.1 aggregationMethod = min [max] pattern = \.max$ xFilesFactor = 0.1 aggregationMethod = max [sum] pattern = \.count$ xFilesFactor = 0 aggregationMethod = sum [default_average] pattern = .* xFilesFactor = 0.5 aggregationMethod = average
You do not really need to change the default config, but is very important to
understand what the config controls and what implications that it has. Graphite
does rollups as part of the metric ingestion according to the rules defined in
storage-schemas.conf. For example, given storage schema rule
when aggregating 6 values (each representing 10 seconds) into a 1min bucket graphite
will use an
aggregationMethod like for example
average. What method to use
will be determined by the rules specified in
The default rules all look at the metric path ending. Does it end with
.count then use
sum when doing rollups, does it end with
max then use
max function, and if it does not
end with max, min or count then use average. This means that naming metrics is very important! But don’t worry if you use StatsD it will send the correct names to graphite.
Lets install supervisord and let it start carbon.
apt-get install supervisor
Create a new file in
/etc/supervisor/conf.d/carbon.conf with the following:
[program:carbon-cache] command = /opt/graphite/bin/carbon-cache.py --debug start stdout_logfile = /var/log/supervisor/%(program_name)s.log stderr_logfile = /var/log/supervisor/%(program_name)s.log autorestart = true stopsignal = QUIT
Graphite api is a light weight version of graphite-web with only the API component (no web ui). It is dead simple to install.
pip install gunicorn graphite-api
You should now have a graphite-api daemon running with an open HTTP API port of 8888.
Create a file
/etc/graphite-api.yaml with an editor and set it’s content to:
search_index: /opt/graphite/storage/index finders: - graphite_api.finders.whisper.WhisperFinder functions: - graphite_api.functions.SeriesFunctions - graphite_api.functions.PieFunctions whisper: directories: - /opt/graphite/storage/whisper time_zone: UTC
Lets create a supervisor file for graphite-api at
[program:graphite-api] command = gunicorn -b 0.0.0.0:8888 -w 2 --log-level info graphite_api.app:app stdout_logfile = /var/log/supervisor/%(program_name)s.log stderr_logfile = /var/log/supervisor/%(program_name)s.log autorestart = true stopsignal = QUIT
A carbon-cache daemon and graphite-api should now be running. Type
supervisorctl status to verify that they are running. You can
http://your_server_ip:8888/metrics/find?query?* in your browser. You should see a json snippet.
cd /tmp/ wget https://grafanarel.s3.amazonaws.com/builds/grafana_2.1.1_amd64.deb sudo dpkg -i grafana_2.1.1_amd64.deb sudo service grafana-server start
Grafana should now be running with default config on port 3000.
Grafana - first steps
Add data source
Open http://your_server_ip:3000 in your browser and login with the default user and password (
- Click on
Data Sourceson the side menu.
- Click on
Add newin the top menu
- Specify name
graphiteand check the
- Specify URL
Your first dashboard
- Click on
- Click on
Homebutton in the top menu, this should open the dashboard search dropdown
- Click on
Newbutton in the bottom of this dropdown
Add a graph
- Click on the green icon to the left to open the row menu
Graphfrom the row menu
- An empty graph panel should appear with title
no title (click here). Click on this title and then
- This will open the graph in edit mode and take you to the metrics tab.
- There is one query already added (assigned letter A) but it is empty.
- Click on
select metricto pick the first graphite metric node. A new
select metriclink will appear until you reached a leaf node.
- Try picking the metric paths for
carbon.agents.<server name>.cpuUsage, you should now see a line appear in the graph!
Writing metrics to Graphite
Graphite has the simplest metric write protocol imaginable. Something that has surely contributed to its wide adoption by metric frameworks and numerous integrations.
prod.server1.requests.count 10 1398969187 <metric.name.and.path> <metric value> <unix_epoch_time_stamp_in_seconds>
There are hundreds of tools and instrumentation frameworks that can send metrics using this protocol.
StatsD is a metrics aggregation daemon that makes it easy for apps on many machines to send measurements like timings and counters and have them aggregated or percentiles calculated.