HTTP API

MetricsGraphiteHTTP API

The HTTP API is the same as that of Graphite, with the addition of ingestion, authentication and meta tags.

First of all, there are two endpoints you will be talking to. They are provided on your grafana.com Hosted Metrics instance details page.

They will look something like:

  • <base_in> : https://something.grafana.net/metrics
  • <base_out> : https://something.grafana.net/graphite

Furthermore, you will need to provision API keys to talk to the API. Each key will be of one of these types:

  • Viewer
  • MetricsPublisher
  • Editor
  • Admin

A “Viewer” key can only be used to execute queries, while a “MetricsPublisher” key can only be used to push metrics into the platform. “Editor” and “Admin” keys can be used for both pushing metrics and performing queries.

API keys can be provisioned on your Grafana.com organization page under “Security > API Keys”.

Authentication

Authentication differs depending on whether your instance is provisioned on a dedicated or a shared cluster. How to tell the difference? On your Grafana.com instance details page, if your query and metrics endpoint look generic like this:

https://graphite-us-central1.grafana.net/graphite
https://graphite-us-central1.grafana.net/metrics

Then you are on a shared cluster. However if your URL’s look more like this:

https://tsdb-123-your-company-name.hosted-metrics.grafana.net/graphite
https://tsdb-123-your-company-name.hosted-metrics.grafana.net/metrics

Then you are on a dedicated cluster.

You have several ways to authenticate:

  • For dedicated clusters, any of these HTTP headers will work:
Authorization: Basic base64(api_key:<api key>)
Authorization: Bearer <api key>
Authorization: Bearer api_key:<api key>
  • For shared clusters, use any of these HTTP headers:
Authorization: Basic base64(<instance id>:<api key>)
Authorization: Bearer <instance id>:<api key>

Note that you can find the instance ID as the username in the “Using Grafana with Hosted Metrics” section of your Grafana.com instance details page

So essentially you can use basic auth with username “api_key” (for dedicated clusters) or your instance ID (for shared clusters) and password the api key that you provisoned

You can also use a bearer token in “username:password” format (if username not specified, “api_key” is assumed)

Common Request Parameters

Many of the API methods involve using Graphite patterns (queries), tag queries and the standard Graphite from/to syntax.

Graphite Patterns

Graphite patterns are queries that involve glob patterns (*, {}, [], ?).

Tag Expressions

Tags expressions are strings, and may have the following formats:

tag=spec    tag value exactly matches spec
tag!=spec   tag value does not exactly match spec
tag=~value  tag value matches the regular expression spec
tag!=~spec  tag value does not match the regular expression spec

Any tag spec that matches an empty value is considered to match series that don’t have that tag, and at least one tag spec must require a non-empty value. Regular expression conditions are treated as being anchored at the start of the value.

From/To (Until)

Graphite from/to

Endpoints

Adding New Data: Posting To /metrics

The main entry point for any publisher to publish data to, be it carbon-relay-ng, or any other script or application such as the hosted-metrics-sender-example

  • Method: POST
  • API key type: any (including Viewer)

Headers

  • Authorization: header is required (see authentication section above)
  • Content-Type: supports 3 values:
    • application/json: the simplest one, and the one used here
    • rt-metric-binary: same datastructure, but messagepack encoded. (see the MetricData Marshal/Encode methods)
    • rt-metric-binary-snappy: same as above, but snappy compressed.

Data Format

Each metricpoint message can have the following properties:

name     // graphite style name (required)
interval // the resolution of the metric in seconds (required)
value    // float64 value (required)
time     // unix timestamp in seconds (required)
tags     // list of key=value pairs of tags (optional)

Example

timestamp_now_rounded=$(($(date +%s) / 10 * 10))
timestamp_prev_rounded=$((timestamp_now_rounded - 10))

curl -X POST -H "Authorization: Bearer $key" -H "Content-Type: application/json" "$base_in" -d '[{
    "name": "test.metric",
    "interval": 10,
    "value": 12.345,
    "time": '$timestamp_prev_rounded'
},
{
    "name": "test.metric",
    "interval": 10,
    "value": 12.345,
    "time": '$timestamp_now_rounded'
},
{
    "name": "test.metric.tagged",
    "interval": 10,
    "value": 1,
    "tags": ["foo=bar", "baz=quux"],
    "time": '$timestamp_prev_rounded'
},
{
    "name": "test.metric.tagged",
    "interval": 10,
    "value": 2,
    "tags": ["foo=bar", "baz=quux"],
    "time": '$timestamp_now_rounded'
}
]'

Deleting Metrics

Non-tagged With /metrics/delete

Deletes metrics which match the query and all child nodes.

Note that unlike the find and render patterns, these queries are recursive. So if the delete query matches a branch, every series under that branch will be deleted. Note that the data stays in the datastore until it expires. Should the metrics enter the system again with the same metadata, the data will show up again.

  • Method: POST
  • API key type: Editor
Parameters
  • user name: api_key
  • password: Your Grafana.com API Key
  • query (required): [Graphite pattern] (#graphite-patterns)
Example
curl -u "api_key:<Your Grafana.com API Key>" https://$base_out/metrics/delete -d query=some.series.to.delete.*

Tagged With /tags/delSeries

Deletes metrics which match the path parameter(s). Note that the data stays in the datastore until it expires. Should the metrics enter the system again with the same metadata, the data will show up again.

  • Method: POST
  • API key type: Editor
Parameters
  • user name: api_key
  • password: Your Grafana.com API Key
  • path (required, multiple allowed): A single Graphite series
Example
curl -u "api_key:<Your Grafana.com API Key>" https://$base_out/tags/delSeries -d "path=some.series;key=value" -d "path=another.series;tag=value"

Finding Metrics

Non-tagged with /metrics/expand

Returns metrics which match the query.

  • Method: GET or POST
  • API key type: any (including MetricsPublisher)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • query: required and may be specified multiple times, in the format described in Graphite pattern
  • groupByExpr: if true, then the results get grouped by the query which yielded them, otherwise all results are in a flat list. (defaults to false)
  • leavesOnly: if true, only leaf nodes get returned, if false branch nodes also get returned. (defaults to false)
  • jsonp: true/false: enables jsonp
Example
curl -H "Authorization: Bearer $key" "$base_out/metrics/expand?groupByExpr=true&query=some.id.of.a.metric.[1-3]&query=some.id.of.a.metric.[67]"
{
    "some.id.of.a.metric.[1-3]": [
        "some.id.of.a.metric.1",
        "some.id.of.a.metric.2",
        "some.id.of.a.metric.3"
    ],
    "some.id.of.a.metric.[67]": [
        "some.id.of.a.metric.6",
        "some.id.of.a.metric.7"
    ]
}

Non-tagged With /metrics/find

Returns metrics which match the query and have received an update since from, de-duplicated by the last name node (used for the name auto-complete).

  • Method: GET or POST
  • API key type: any (including MetricsPublisher)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • query: required and may be specified multiple times, in the format described in Graphite pattern
  • format: json, treejson, completer, pickle, or msgpack. (defaults to json)
  • jsonp: true/false: enables jsonp
  • from: Graphite from time specification (defaults to now-24hours)
Output formats
  • json, treejson (default/unspecified): the standard format
  • completer: used for graphite-web’s completer UI
  • msgpack: optimized transfer format
  • pickle: deprecated
Example
curl -H "Authorization: Bearer $key" "$base_out/metrics/find?query=metrictank"
[
    {
        "allowChildren": 1,
        "expandable": 1,
        "leaf": 0,
        "id": "metrictank",
        "text": "metrictank",
        "context": {}
    }
]

The response indicates that there are metric names that live under the “metrictank” term (it is expandable) and there is no data under the name “metrictank” (it is not a leaf node).

So we update the query to see what we can expand to:

curl -H "Authorization: Bearer $key" "$base_out/metrics/find?query=metrictank.*"
[
    {
        "allowChildren": 1,
        "expandable": 1,
        "leaf": 0,
        "id": "metrictank.aggstats",
        "text": "aggstats",
        "context": {}
    }
]

The response for the updated query shows which data lives under the “metrictank” name, in this case the tree extends under “metrictank.aggstats”.

As we continue to dig deeper into the tree, by updating our query based on what we get back, we eventually end up at the leaf:

curl -H "Authorization: Bearer $key" "$base_out/metrics/find?query=metrictank.aggstats.*.tank.metrics_active.gauge32"
[
    {
        "allowChildren": 0,
        "expandable": 0,
        "leaf": 1,
        "id": "metrictank.aggstats.us-east2-id-name.tank.metrics_active.gauge32",
        "text": "gauge32",
        "context": {}
    }
]

Tagged With /tags/findSeries

Returns metrics which match tag queries and have received an update since from. Note: the returned results are not deduplicated and in certain cases it is possible that duplicate entries will be returned.

  • Method: GET or POST
  • API key type: any (including MetricsPublisher)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • expr (required): a list of tag expressions
  • from: Graphite from time specification (optional. defaults to now-24hours)
  • format: series-json, lastts-json. (defaults to series-json)
  • limit: max number to return. (default: 0) Note: the resultset is also subjected to the cluster configuration. if the result set is larger than the cluster configuration, an error is returned. If it breaches the provided limit, the result is truncated.
  • meta: If false and format is series-json then return series names as array (graphite compatibility). If true, include meta information like warnings. (defaults to false)
Example
curl -H "Authorization: Bearer $key" "$base_out/tags/findSeries?expr=datacenter=dc1&expr=server=web01"

[
  "disk.used;datacenter=dc1;rack=a1;server=web01"
]
curl -H "Authorization: Bearer $key" "$base_out/tags/findSeries?expr=datacenter=dc1&expr=server=web01&format=lastts-json"

{
    "series": [
        {
            "lastTs": 1576683990,
            "val": "disk.used;datacenter=dc1;rack=a1;server=web01"
        }
    ]
}

Tag Exploration

Count tag values With /tags/terms

Returns count of series for each tag value which matches tag queries for a given set of tag keys.

  • Method: GET or POST
  • API key type: any (including MetricsPublisher)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • expr (required): a list of tag expressions
  • tags: a list of tag keys for which to count values series
Example
curl -H "Authorization: Bearer $key" "$base_out/tags/terms?expr=datacenter=dc1&expr=server=web01&tags=rack"

{
  "totalSeries": 5892,
  "terms": {
    "rack": {
      "a1": 2480,
      "a2": 465,
      "b1": 2480,
      "b2": 467
    }
  }
}

Render /render (return data for a given query)

Graphite-web-like api. It can return JSON, pickle or messagepack output

  • Method: GET or POST (recommended. as GET may result in too long URL’s)
  • API key type: any (viewer, publisher, editor)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • maxDataPoints: int (default: 800)
  • target: mandatory. one or more metric names or patterns.
  • from: see timespec format (default: 24h ago) (exclusive)
  • to/until: see timespec format(default: now) (inclusive)
  • format: json, msgp, pickle, or msgpack (default: json) (note: msgp and msgpack are similar, but msgpack is for use with graphite)
  • meta: Use ‘meta=true’ to enable performance and series lineage metadata in the response body. See below for more information.

Data queried for must be stored under the given org or be public data (see multi-tenancy)

Example

curl -H "Authorization: Bearer $key" "http://localhost:6060/render?target=statsd.fakesite.counters.session_start.*.count&from=3h&to=2h"

Metadata

The metadata of a render response (provided when meta=true is passed), includes:

  • Response global performance measurements
  • Series-specific lineage information describing storage-schemas, read archive, archive interval, and any consolidation and normalization applied. Explicit function calls like summarize are not considered runtime consolidation for this purpose.

This metadata is shown in Grafana’s panel inspector, provided your Graphite data source has the “type” selector set to Metrictank.

Response-global performance measurements
Key Description
executeplan.resolve-series.ms Time spent doing the (distributed) index query
executeplan.get-targets.ms Time spent doing the (distributed) fetching of data
executeplan.prepare-series.ms Time spent preparing for plan run. so merging, sorting of fetched series
executeplan.plan-run.ms Time spent executing all processing functions
executeplan.series-fetch.count Number of series fetched
executeplan.points-fetch.count Number of points fetched
executeplan.points-return.count Number of points returned
executeplan.cache-miss.count Number of cache misses (series with no useful chunks in cache)
executeplan.cache-hit-partial.count Number of partial cache hits (series with some useful chunks in cache)
executeplan.cache-hit.count Number of full cache hits (series with all needed chunks in cache)
executeplan.chunks-from-tank.count Number of chunks loaded from tank
executeplan.chunks-from-cache.count Number of chunks loaded from chunk cache
executeplan.chunks-from-store.count Number of chunks loaded from data storage
Series-specific lineage information

Every output series comes with lineage information. The lineage information is one or more lineage sections. Each section describes a certain lineage, and the number of input series matching that lineage that were part of the corresponding output series. Each lineage section has these fields:

Key Description
schema-name Name of the section in storage-schemas.conf
schema-retentions Retentions defined in storage-schemas.conf
archive-read Which archive was read as defined in the retentions. (0 means raw, 1 first rollup, etc)
archive-interval The native interval of the archive that was read
aggnum-norm If >1, number of points aggregated together per point, as part of normalization
aggnum-rc If >1, number of points aggregated together per output point, as part of runtime consolidation (MaxDataPoints)
consolidator-normfetch Consolidator used for normalization (if aggnum-norm > 1) and which rollup was read (if archive-read > 0)
consolidator-rc Consolidator used for runtime consolidation (MaxDataPoints) (if aggnum-rc > 1)
count Number of input series matching this lineage that were part of this output series