Instrument and send data

Metrics

Send or visualize Graphite metrics

Graphite HTTP API

Grafana Cloud

Graphite HTTP API

The HTTP API is the same as that of Graphite, with the addition of ingestion, authentication, and storage-aggregation/storage-schemas.conf endpoints.

First of all, there are two endpoints you will be talking to. They are provided in the Cloud Portal by first going to the Details for your stack, then selecting the Details of the Graphite card within your stack.

They will look something like:

<ingest_endpoint> : https://something.grafana.net/graphite/metrics
<query_endpoint> : https://something.grafana.net/graphite

You also need to create an access policy with the metrics:read and/or metrics:write scopes. You can then create a token for that access policy, which will be used to talk to the API (as shown below). See Cloud Access Policies for more information.

Authentication

All endpoints require authentication, which can done using any of these HTTP headers:

Authorization: Basic base64(<user>:<token>)
Authorization: Bearer <user>:<token>

Your user can be found in the details of your Graphite instance details page in your Grafana.com stack.

Common Request Parameters

Many of the API methods involve using Graphite patterns (queries), tag queries and the standard Graphite from/to syntax.

Graphite Patterns

Graphite patterns are queries that involve glob patterns (*, {}, [], ?).

Tag Expressions

Tags expressions are strings, and may have the following formats:

tag=spec    tag value exactly matches spec
tag!=spec   tag value does not exactly match spec
tag=~value  tag value matches the regular expression spec
tag!=~spec  tag value does not match the regular expression spec

Any tag spec that matches an empty value is considered to match series that don’t have that tag, and at least one tag spec must require a non-empty value. Regular expression conditions are treated as being anchored at the start of the value.

From/To (Until)

Graphite from/to

Endpoints

Adding New Data: Posting To `/metrics`

The main entry point for any publisher to publish data to, be it carbon-relay-ng, or any other script or application such as the hosted-metrics-sender-example

Method: POST
Access policy scope: metrics:write

Headers

Content-Type: supports 3 values:
- application/json: the simplest one, and the one used here
- rt-metric-binary: same datastructure, but messagepack encoded
- rt-metric-binary-snappy: same as above, but snappy compressed

Data Format

Each metricpoint message can have the following properties:

name     // Graphite style name (required)
interval // the resolution of the metric in seconds (required)
value    // float64 value (required)
time     // unix timestamp in seconds (required)
tags     // list of key=value pairs of tags (optional)

Example

timestamp_now_rounded=$(($(date +%s) / 10 * 10))
timestamp_prev_rounded=$((timestamp_now_rounded - 10))

curl -X POST -H "Authorization: Bearer <user>:<token>" -H "Content-Type: application/json" "<ingest_endpoint>" -d '[{
    "name": "test.metric",
    "interval": 10,
    "value": 12.345,
    "time": '$timestamp_prev_rounded'
},
{
    "name": "test.metric",
    "interval": 10,
    "value": 12.345,
    "time": '$timestamp_now_rounded'
},
{
    "name": "test.metric.tagged",
    "interval": 10,
    "value": 1,
    "tags": ["foo=bar", "baz=quux"],
    "time": '$timestamp_prev_rounded'
},
{
    "name": "test.metric.tagged",
    "interval": 10,
    "value": 2,
    "tags": ["foo=bar", "baz=quux"],
    "time": '$timestamp_now_rounded'
}
]'

Render `/render` (return data for a given query)

Graphite-web-like api. It can return JSON, pickle or messagepack output. https://graphite.readthedocs.io/en/latest/render_api.html

Method: GET or POST (recommended, as GET may result in too long URLs)
Access policy scope: metrics:read

Parameters

maxDataPoints: unsigned int (default: 800)
target (required): one or more metric names or patterns.
from: see timespec format (default: 24h ago) (exclusive)
to/until: see timespec format(default: now) (inclusive)
format: json, msgp, pickle, or msgpack (default: json) (note: msgp and msgpack are similar, but msgpack is for use with Graphite)
meta: This is a legacy flag used for a previous version of Cloud Graphite. For supported functions, the response will look like { "meta": {...}, "series" : <default json format> }. If a function does not support the meta parameter, the reponse will be returned in the default json format instead. The recommendation is to avoid using the meta parameter, as many of the metadata fields are not set by the current Cloud Graphite version and this parameter may be completely removed in later versions.

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/render?target=statsd.fakesite.counters.session_start.*.count&from=-3h&to=-2h"

Response

https://graphite.readthedocs.io/en/latest/render_api.html

Finding Metrics

Non-tagged with `/metrics/expand`

Returns metrics which match the query. Results are limited to series that have received an update no longer than 32 days ago for performance reasons.

Method: GET or POST
Access policy scope: metrics:read

Parameters

query: required and may be specified multiple times, in the format described in Graphite pattern
groupByExpr: if true, then the results get grouped by the query which yielded them, otherwise all results are in a flat list. (defaults to false)
leavesOnly: if true, only leaf nodes get returned, if false branch nodes also get returned. (defaults to false)
jsonp: true/false: enables jsonp

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/expand?groupByExpr=true&query=some.id.of.a.metric.[1-3]&query=some.id.of.a.metric.[67]"
{
    "some.id.of.a.metric.[1-3]": [
        "some.id.of.a.metric.1",
        "some.id.of.a.metric.2",
        "some.id.of.a.metric.3"
    ],
    "some.id.of.a.metric.[67]": [
        "some.id.of.a.metric.6",
        "some.id.of.a.metric.7"
    ]
}

Non-tagged With `/metrics/find`

Returns metrics which match the query and have received an update since from, de-duplicated by the last name node (used for the name auto-complete). Results might be limited to data that is no older than 32 days for performance reasons.

Method: GET or POST
Access policy scope: metrics:read

Parameters

query: required and may be specified multiple times, in the format described in Graphite pattern
format: json, treejson, completer, pickle, or msgpack. (defaults to json)
jsonp: true/false: enables jsonp
from: see timespec format (defaults to now-24hours)

Output formats

json, treejson (default/unspecified): the standard format
completer: used for graphite-web’s completer UI
msgpack: optimized transfer format
pickle: deprecated

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/find?query=foobar"
[
    {
        "allowChildren": 1,
        "expandable": 1,
        "leaf": 0,
        "id": "foobar",
        "text": "foobar",
        "context": {}
    }
]

The response indicates that there are metric names that live under the “foobar” term (it is expandable) and there is no data under the name “foobar” (it is not a leaf node).

So we update the query to see what we can expand to:

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/find?query=foobar.*"
[
    {
        "allowChildren": 1,
        "expandable": 1,
        "leaf": 0,
        "id": "foobar.aggstats",
        "text": "aggstats",
        "context": {}
    }
]

The response for the updated query shows which data lives under the “foobar” name, in this case the tree extends under “foobar.aggstats”.

As we continue to dig deeper into the tree, by updating our query based on what we get back, we eventually end up at the leaf:

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/find?query=foobar.aggstats.*.tank.metrics_active.gauge32"
[
    {
        "allowChildren": 0,
        "expandable": 0,
        "leaf": 1,
        "id": "foobar.aggstats.us-east2-id-name.tank.metrics_active.gauge32",
        "text": "gauge32",
        "context": {}
    }
]

Tagged With `/tags/findSeries`

Returns metrics which match tag queries and have received an update since from. Note: the returned results are not deduplicated and in certain cases it is possible that duplicate entries will be returned.

Method: GET or POST
Access policy scope: metrics:read

Parameters

expr (required): a list of tag expressions
from: see timespec format (defaults to now-24hours)
format: series-json, lastts-json. (defaults to series-json)
limit: max number to return. Zero means no limit. (default: 0) Note: the resultset is also subjected to the cluster configuration. if the result set is larger than the cluster configuration, an error is returned. If it breaches the provided limit, the result is truncated.
meta: this is a legacy flag used for a previous version of Cloud Graphite. If enabled, the response includes meta information like warnings, with the format { "warning": {...}, "series" : <default format> }. The recommendation is to avoid using the meta parameter, as this parameter may be completely removed in later versions. (defaults to false)

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/findSeries?expr=datacenter=dc1&expr=server=web01"
[
  "disk.used;datacenter=dc1;rack=a1;server=web01"
]

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/findSeries?expr=datacenter=dc1&expr=server=web01&format=lastts-json"
{
    "series": [
        {
            "lastTs": 1576683990,
            "val": "disk.used;datacenter=dc1;rack=a1;server=web01"
        }
    ]
}

All Series With `/metrics/index.json`

Returns the names of all metrics (both tagged and non-tagged) that have received an update within the last hour.

Method: GET or POST
Access policy scope: metrics:read

Parameters

jsonp: a jsonp callback to wrap the response (optional)

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/index.json
[
    "collectd.prod-us-central-0.mysql.bytes.ibuf_size",
    "collectd.prod-eu-west-2.mysql.bytes.ibuf_size",
    "collectd.prod-us-east-2.mysql.bytes.ibuf_size"
]

Exploring Tags

Finding Tags With `/tags`

Returns the existing metric tags that match the given filter and time range. Results might be limited to data that is no older than 32 days for performance reasons.

Method: GET or POST
Access policy scope: metrics:read

Parameters

from: see timespec format (defaults to now-24hours)
to/until: see timespec format(default: now) (inclusive)
filter: a regex to use to filter the tags. (default: .+)

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags?filter=foo"

[
  {
    "tag": "foobar"
  },
  {
    "tag": "foobaz"
  }
]

Finding Tag Values With `/tags/{tag}`

Returns the existing values for the given metric tag. Results might be limited to data that is no older than 32 days for performance reasons.

Method: GET or POST
Access policy scope: metrics:read

Parameters

from: see timespec format (defaults to now-24hours)
to/until: see timespec format (default: now) (inclusive)
tag (required): the tag for which to find values.
filter: a regex to use to filter the values. (default: .+)

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/foobar?filter=gr"

{
  "tag": "foobar",
  "values": [
    {
      "count": 1,
      "value": "graphite"
    },
    {
      "count": 2,
      "value": "grafana"
    }
  ]
}

Autocompletion Of Tags With `/tags/autoComplete/tags`

Returns the existing tags of metrics that match the given expression. Results might be limited to data that is no older than 32 days for performance reasons.

Method: GET or POST
Access policy scope: metrics:read

Parameters

from: see timespec format (defaults to now-24hours)
to/until: see timespec format(default: now) (inclusive)
expr: an expression to filter the tagged metrics. Multiple are supported.
tagPrefix: a prefix to filter tags further. (default: “”)
limit: a number to limit to number of results. (default: 100)

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/autoComplete/tags?expr=foobar=graphite&tagPrefix=qu"

[
  "qux",
  "quy"
]

Autocompletion Of Tag Values With `/tags/autoComplete/values`

Returns the existing values for tags of metrics that match the given expression. Results might be limited to data that is no older than 32 days for performance reasons.

Method: GET or POST
Access policy scope: metrics:read

Parameters

from: see timespec format (defaults to now-24hours)
to/until: see timespec format(default: now) (inclusive)
tag (required): the tag for which to find values.
expr: an expression to filter the tagged metrics. Multiple are supported.
valuePrefix: a prefix to filter the tag’s values further. (default: “”)
limit: a number to limit to number of results. (default: 100)

Example

curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/autoComplete/values?tag=qux&expr=foobar=graphite&valuePrefix=lo"

[
  "loremipsum"
]

Read or Adjust storage-schemas.conf and storage-aggregation.conf

Grafana Cloud Graphite allows for dynamic changing of both the Graphite storage schemas and aggregation configurations. When interval data or rollups are improperly configured (for example, defaults not matching the real data or their intended use) this can result in suboptimal visualizations. Correcting these configurations can fix these problems instantly, unlike with standard Graphite installations, because changes are applied retroactively without any loss of information. You can post as many changes as needed. If you do not update these configurations, the default Grafana Cloud settings are applied instead.

Carbon-relay-NG version 1.2 or later automatically pushes both of these files for you. You should not need to manually push new configurations, and carbon-relay-ng is the recommended way of publishing these files.

You can manually POST to this endpoint when:

you haven’t upgraded your relay yet.
you want to experiment iteratively without having to repeatedly restart your relay.

There are two URL paths:

/graphite/config/storageSchemas for storage-schemas.conf
/graphite/config/storageAggregation for storage-aggregation.conf
Method: GET or POST

Parameters

None, and when posting the body is the file contents.

Example

You can download the current Schema and Aggregation configurationss using the API endpoints in GET mode:

curl -X GET -H "Authorization: Bearer <user>:<api key>" <query_endpoint>/config/storageSchemas
curl -X GET -H "Authorization: Bearer <user>:<api key>" <query_endpoint>/config/storageAggregation

You can push new configurations in POST mode:

curl -H "Authorization: Bearer <user>:<api key>" "<query_endpoint>/config/storageSchemas" --data-binary '@<path-to-storage-schemas.conf>'
curl -H "Authorization: Bearer <user>:<api key>" "<query_endpoint>/config/storageAggregation" --data-binary '@<path-to-storage-aggregation.conf>'

Feedback

Graphite HTTP API

Authentication

Common Request Parameters

Graphite Patterns

Tag Expressions

From/To (Until)

Endpoints

Adding New Data: Posting To /metrics

Headers

Data Format

Example

Render /render (return data for a given query)

Parameters

Example

Response

Finding Metrics

Non-tagged with /metrics/expand

Parameters

Example

Non-tagged With /metrics/find

Parameters

Output formats

Example

Tagged With /tags/findSeries

Parameters

Example

All Series With /metrics/index.json

Parameters

Example

Exploring Tags

Finding Tags With /tags

Parameters

Example

Finding Tag Values With /tags/{tag}

Parameters

Example

Autocompletion Of Tags With /tags/autoComplete/tags

Parameters

Example

Autocompletion Of Tag Values With /tags/autoComplete/values

Parameters

Example

Read or Adjust storage-schemas.conf and storage-aggregation.conf

Parameters

Example

Was this page helpful?

Related documentation

Related resources from Grafana Labs

Adding New Data: Posting To `/metrics`

Render `/render` (return data for a given query)

Non-tagged with `/metrics/expand`

Non-tagged With `/metrics/find`

Tagged With `/tags/findSeries`

All Series With `/metrics/index.json`

Finding Tags With `/tags`

Finding Tag Values With `/tags/{tag}`

Autocompletion Of Tags With `/tags/autoComplete/tags`

Autocompletion Of Tag Values With `/tags/autoComplete/values`