MetricsGraphite

Metrics — Graphite

Grafana Labs’s Metrics service offers a graphite-compatible monitoring backend-as-a-service. It behaves like a regular Graphite datasource within Grafana (or other tools), but behind the scenes, it is a sophisticated platform run by a team of dedicated engineers.

NOTE: Graphite is stable, but older and receives only infrequent updates. If you are already using it, these docs should be helpful; make sure when you read the official Graphite docs that you read the version specific to the release you are using. If you are deciding what to use, we recommend Prometheus.

The main Graphite docs are the canonical source for information about Graphite. Graphite query functions are supported and available in the Grafana UI, whether using the Grafana Explore feature or writing queries for panels. The caveat is that functions that only exist in later Graphite releases will not be available if your Graphite installation is an older release.

Finding your API endpoints

Several examples below have a <instance URL> placeholder. To identify your instance URL, login at grafana.com and navigate to your Metrics instance details.

Sending Data with Carbon-Relay-NG

There are a variety of ways that you can send your data to Metrics.

In most situations, you should install a carbon-relay-ng service in each of the datacenter or regions that you will be sending metrics from.

This will accept plain-text carbon (Graphite) input, and stream your encrypted metrics to the Grafana Cloud Metrics. Since carbon-relay-ng can buffer metric streams in memory, this also provides increased resiliency to connectivity issues.

carbon-relay-ng is available for most Linux platforms, and is easy to install.

We recommend installing the latest version. Once installed, you can configure the relay in two steps:

1. Edit carbon-relay-ng.conf

Edit the carbon-relay-ng.conf configuration file (normally located at /etc/carbon-relay-ng/carbon-relay-ng.conf), and replace it with the settings below:

  • The “apikey” setting must be a Grafana.com API Key with the editor or admin role.
  • Make sure the “schemasFile” field is set to the path to your storage-schemas.conf file (see below)
## Global settings ##
# instance id's distinguish stats of multiple relays.
# do not run multiple relays with the same instance id.
# supported variables:
#  ${HOST} : hostname
instance = "${HOST}"

## System ##
# this setting can be used to override the default GOMAXPROCS logic
# it is ignored if the GOMAXPROCS environment variable is set
max_procs = 2
pid_file = "carbon-relay-ng.pid"
# directory for spool files
spool_dir = "spool"

## Logging ##
# one of trace debug info warn error fatal panic
# see docs/logging.md for level descriptions
# note: if you used to use "notice", you should now use "info".
log_level = "info"

## Inputs ##
### plaintext Carbon ###
listen_addr = "0.0.0.0:2003"
# close inbound plaintext connections if they've been idle for this long ("0s" to disable)
plain_read_timeout = "2m"
### Pickle Carbon ###
pickle_addr = "0.0.0.0:2013"
# close inbound pickle connections if they've been idle for this long ("0s" to disable)
pickle_read_timeout = "2m"

## Validation of inputs ##
# you can also validate that each series has increasing timestamps
validate_order = false

# How long to keep track of invalid metrics seen
# Useful time units are "s", "m", "h"
bad_metrics_max_age = "24h"

[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = 'https://<instance URL>.hosted-metrics.grafana.net/metrics'
apikey = '<Your Grafana.com API Key>'
schemasFile = '/etc/carbon-relay-ng/storage-schemas.conf'

## Instrumentation ##
[instrumentation]
# in addition to serving internal metrics via expvar, you can send them to graphite/carbon
# IMPORTANT: setting this to "" will disable flushing, and metrics will pile up and lead to OOM
# see https://github.com/graphite-ng/carbon-relay-ng/issues/50
# so for now you MUST send them somewhere. sorry.
# (Also, the interval here must correspond to your setting in storage-schemas.conf if you use grafana hosted metrics)
graphite_addr = "localhost:2003"
graphite_interval = 10000  # in ms

2. Create storage-schemas.conf

Create a new /etc/carbon-relay-ng/storage-schemas.conf file. This file has the same format as the Graphite storage-schemas.conf file.

You can paste the below into the file to get started:

[crng-service]
  pattern = ^service_is_carbon-relay-ng
  # interval should match graphite_interval in your relay configuration (default 10.000 ms)
  retentions = 10s:1d
[crng-stats]
  pattern = ^carbon-relay-ng\.stats
  # interval should match graphite_interval in your relay configuration (default 10.000 ms)
  retentions = 10s:1d
[default]
  pattern = .*
  retentions = 10s:1d

This default assumes you are sending metrics at a 10s resolution, ie. 6 times per minute, which you may need to change. It is important that this file accurately describes your metrics and their raw resolutions (retentions mentioned in this file are ignored) You may need to extend it to match your metrics, or alternatively, copy it from your Graphite server and add in the rules for carbon-relay-ng metrics. If you update the carbon-relay-ng ‘graphite_interval’ parameter to something other than the default of 10.000ms (10s), you should set that update the rules for carbon-relay-ng here as well.

Once configured, you can send metrics to the relay in carbon/Graphite format in port 2003, and they should show up in Grafana.