Graphite metrics

Grafana Labs’s metrics service offers a graphite-compatible monitoring backend-as-a-service. It behaves like a regular Graphite datasource within Grafana (or other tools), but behind the scenes, it is a sophisticated platform run by a team of dedicated engineers.

Graphite data ingestion
Graphite HTTP API
Graphite FAQ

NOTE: Graphite is stable, but older and receives only infrequent updates. If you are already using it, these docs should be helpful; make sure when you read the official Graphite docs that you read the version specific to the release you are using. If you are deciding what to use, we recommend Prometheus.

The main Graphite docs are the canonical source for information about Graphite. Graphite query functions are supported and available in the Grafana UI, whether using the Grafana Explore feature or writing queries for panels. The caveat is that functions that only exist in later Graphite releases will not be available if your Graphite installation is an older release.

Finding your API endpoints

There are two endpoints you will be talking to. They are provided on your grafana.com Hosted Metrics instance details page.

They will look something like:

<ingest_endpoint> : https://something.grafana.net/metrics
<query_endpoint> : https://something.grafana.net/graphite

Sending Data with Carbon-Relay-NG

There are a variety of ways that you can send your data to Metrics.

In most situations, you should install a carbon-relay-ng service in each of the datacenter or regions that you will be sending metrics from.

This will accept plain-text carbon (Graphite) input, and stream your encrypted metrics to the Grafana Cloud Metrics. Since carbon-relay-ng can buffer metric streams in memory, this also provides increased resiliency to connectivity issues.

carbon-relay-ng is available for most Linux platforms, and is easy to install.

Version 1.2 or higher is recommended as it helps in configuring your schemas and aggregation.

Once installed, you can configure the relay in two steps:

1. Edit carbon-relay-ng.conf

Edit the carbon-relay-ng.conf configuration file (normally located at /etc/carbon-relay-ng/carbon-relay-ng.conf), and replace it with the settings below:

The “apikey” setting must be a Grafana.com API Key with the editor or admin role.
Make sure the “schemasFile” and “aggregationFile” fields are set to the path to your storage-schemas.conf and storage-aggregation.conf files (see section below)

## Global settings ##
# instance id's distinguish stats of multiple relays.
# do not run multiple relays with the same instance id.
# supported variables:
#  ${HOST} : hostname
instance = "${HOST}"

## System ##
# this setting can be used to override the default GOMAXPROCS logic
# it is ignored if the GOMAXPROCS environment variable is set
max_procs = 2
pid_file = "carbon-relay-ng.pid"
# directory for spool files
spool_dir = "spool"

## Logging ##
# one of trace debug info warn error fatal panic
# see docs/logging.md for level descriptions
# note: if you used to use "notice", you should now use "info".
log_level = "info"

## Inputs ##
### plaintext Carbon ###
listen_addr = "0.0.0.0:2003"
# close inbound plaintext connections if they've been idle for this long ("0s" to disable)
plain_read_timeout = "2m"
### Pickle Carbon ###
pickle_addr = "0.0.0.0:2013"
# close inbound pickle connections if they've been idle for this long ("0s" to disable)
pickle_read_timeout = "2m"

## Validation of inputs ##
# you can also validate that each series has increasing timestamps
validate_order = false

# How long to keep track of invalid metrics seen
# Useful time units are "s", "m", "h"
bad_metrics_max_age = "24h"

[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = '<metrics endpoint>'
apikey = '<Your Grafana.com API Key>'
schemasFile = '/etc/carbon-relay-ng/storage-schemas.conf'
aggregationFile = '/etc/carbon-relay-ng/storage-aggregation.conf' # optional

## Instrumentation ##
[instrumentation]
# in addition to serving internal metrics via expvar, you can send them to graphite/carbon
# IMPORTANT: setting this to "" will disable flushing, and metrics will pile up and lead to OOM
# see https://github.com/grafana/carbon-relay-ng/issues/50
# so for now you MUST send them somewhere. sorry.
# (Also, the interval here must correspond to your setting in storage-schemas.conf if you use grafana hosted metrics)
graphite_addr = "localhost:2003"
graphite_interval = 10000  # in ms

2. Provide storage-schemas.conf and storage-aggregation.conf

The relay will need a storage-schemas.conf file. This describes your metrics resolution (interval) and rollups/retention.

Optionally, you can also provide a storage-aggregation.conf file. This describes using what function the rollups (long term downsampled data) should be aggregated to. For some values you may want to aggregate into averages (default), for others you may want the minima and maxima, etc. If this file is not provided, default Grafana Cloud aggregations will be applied instead.

If you already use Graphite, you can use your existing files as starting points, although you should add rules to cover the carbon-relay-ng stats themselves.

storage-schemas.conf

Here’s a default to get started. Note the additional rules for the carbon-relay-ng stats themselves.

[crng-service]
  pattern = ^service_is_carbon-relay-ng
  # interval should match graphite_interval in your relay configuration (default 10.000 ms)
  retentions = 10s:1y
[crng-stats]
  pattern = ^carbon-relay-ng\.stats
  # interval should match graphite_interval in your relay configuration (default 10.000 ms)
  retentions = 10s:1y
[default]
  pattern = .*
  retentions = 10s:8d,1m:60d,30m:2y

Make sure your rules accurately describe your metrics. The carbon-relay-ng intervals should match the ‘graphite_interval’ config setting. This default assumes you are sending metrics at a 10s resolution, ie. 6 times per minute, which you may need to change.

storage-aggregation.conf

You can simply take your Graphite configuration file or use this to get started:

[default]
  pattern = .*
  xFilesFactor = 0.5
  aggregationMethod = avg

Once these files are configured, you can send metrics to the relay in carbon/Graphite format in port 2003, and they should show up in Grafana.