MetricsGraphiteHTTP API

HTTP API

The HTTP API is the same as that of Graphite, with the addition of ingestion, authentication and meta tags.

First of all, there are two endpoints you will be talking to. They are provided on your grafana.com Hosted Metrics instance details page.

They will look something like:

  • <base_in> : https://something.grafana.net/metrics
  • <base_out> : https://something.grafana.net/graphite

Furthermore, you will need to provision API keys to talk to the API. Each key will be of one of these types:

  • Viewer
  • MetricsPublisher
  • Editor
  • Admin

A “Viewer” key can only be used to execute queries, while a “MetricsPublisher” key can only be used to push metrics into the platform. “Editor” and “Admin” keys can be used for both pushing metrics and performing queries.

API keys can be provisioned on your Grafana.com organization page under “Security > API Keys”.

Authentication

Authentication differs depending on whether your instance is provisioned on a dedicated or a shared cluster. How to tell the difference? On your Grafana.com instance details page, if your query and metrics endpoint look generic like this:

https://graphite-us-central1.grafana.net/graphite
https://graphite-us-central1.grafana.net/metrics

Then you are on a shared cluster. However if your URL’s look more like this:

https://tsdb-123-your-company-name.hosted-metrics.grafana.net/graphite
https://tsdb-123-your-company-name.hosted-metrics.grafana.net/metrics

Then you are on a dedicated cluster.

You have several ways to authenticate:

  • For dedicated clusters, any of these HTTP headers will work:
Authorization: Basic base64(api_key:<api key>)
Authorization: Bearer <api key>
Authorization: Bearer api_key:<api key>
  • For shared clusters, use any of these HTTP headers:
Authorization: Basic base64(<instance id>:<api key>)
Authorization: Bearer <instance id>:<api key>

Note that you can find the instance ID as the username in the “Using Grafana with Hosted Metrics” section of your Grafana.com instance details page

So essentially you can use basic auth with username “api_key” (for dedicated clusters) or your instance ID (for shared clusters) and password the api key that you provisoned

You can also use a bearer token in “username:password” format (if username not specified, “api_key” is assumed)

Common Request Parameters

Many of the API methods involve using Graphite patterns (queries), tag queries and the standard Graphite from/to syntax.

Graphite Patterns

Graphite patterns are queries that involve glob patterns (*, {}, [], ?).

Tag Expressions

Tags expressions are strings, and may have the following formats:

tag=spec    tag value exactly matches spec
tag!=spec   tag value does not exactly match spec
tag=~value  tag value matches the regular expression spec
tag!=~spec  tag value does not match the regular expression spec

Any tag spec that matches an empty value is considered to match series that don’t have that tag, and at least one tag spec must require a non-empty value. Regular expression conditions are treated as being anchored at the start of the value.

From/To (Until)

Graphite from/to

Endpoints

Adding New Data: Posting To /metrics

The main entry point for any publisher to publish data to, be it carbon-relay-ng, or any other script or application such as the hosted-metrics-sender-example

  • Method: POST
  • API key type: any (including Viewer)

Headers

  • Authorization: header is required (see authentication section above)
  • Content-Type: supports 3 values:
    • application/json: the simplest one, and the one used here
    • rt-metric-binary: same datastructure, but messagepack encoded. (see the MetricData Marshal/Encode methods)
    • rt-metric-binary-snappy: same as above, but snappy compressed.

Data Format

Each metricpoint message can have the following properties:

name     // graphite style name (required)
interval // the resolution of the metric in seconds (required)
value    // float64 value (required)
time     // unix timestamp in seconds (required)
tags     // list of key=value pairs of tags (optional)

Example

timestamp_now_rounded=$(($(date +%s) / 10 * 10))
timestamp_prev_rounded=$((timestamp_now_rounded - 10))

curl -X POST -H "Authorization: Bearer $key" -H "Content-Type: application/json" "$base_in" -d '[{
    "name": "test.metric",
    "interval": 10,
    "value": 12.345,
    "time": '$timestamp_prev_rounded'
},
{
    "name": "test.metric",
    "interval": 10,
    "value": 12.345,
    "time": '$timestamp_now_rounded'
},
{
    "name": "test.metric.tagged",
    "interval": 10,
    "value": 1,
    "tags": ["foo=bar", "baz=quux"],
    "time": '$timestamp_prev_rounded'
},
{
    "name": "test.metric.tagged",
    "interval": 10,
    "value": 2,
    "tags": ["foo=bar", "baz=quux"],
    "time": '$timestamp_now_rounded'
}
]'

Tag Exploration

Count tag values With /tags/terms

Returns count of series for each tag value which matches tag queries for a given set of tag keys.

  • Method: GET or POST
  • API key type: any (including MetricsPublisher)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • expr (required): a list of tag expressions
  • tags: a list of tag keys for which to count values series
Example
curl -H "Authorization: Bearer $key" "$base_out/tags/terms?expr=datacenter=dc1&expr=server=web01&tags=rack"

{
  "totalSeries": 5892,
  "terms": {
    "rack": {
      "a1": 2480,
      "a2": 465,
      "b1": 2480,
      "b2": 467
    }
  }
}

Render /render (return data for a given query)

Graphite-web-like api. It can return JSON, pickle or messagepack output

  • Method: GET or POST (recommended. as GET may result in too long URL’s)
  • API key type: any (viewer, publisher, editor)
Headers
  • Authorization: Bearer <api-key> required
Parameters
  • maxDataPoints: int (default: 800)
  • target: mandatory. one or more metric names or patterns.
  • from: see timespec format (default: 24h ago) (exclusive)
  • to/until: see timespec format(default: now) (inclusive)
  • format: json, msgp, pickle, or msgpack (default: json) (note: msgp and msgpack are similar, but msgpack is for use with graphite)
  • meta: Use ‘meta=true’ to enable performance and series lineage metadata in the response body. See below for more information.

Data queried for must be stored under the given org or be public data (see multi-tenancy)

Example

curl -H "Authorization: Bearer $key" "http://localhost:6060/render?target=statsd.fakesite.counters.session_start.*.count&from=3h&to=2h"

Metadata

The metadata of a render response (provided when meta=true is passed), includes:

  • Response global performance measurements
  • Series-specific lineage information describing storage-schemas, read archive, archive interval, and any consolidation and normalization applied. Explicit function calls like summarize are not considered runtime consolidation for this purpose.

This metadata is shown in Grafana’s panel inspector, provided your Graphite data source has the “type” selector set to Metrictank.

Response-global performance measurements
Key Description
executeplan.resolve-series.ms Time spent doing the (distributed) index query
executeplan.get-targets.ms Time spent doing the (distributed) fetching of data
executeplan.prepare-series.ms Time spent preparing for plan run. so merging, sorting of fetched series
executeplan.plan-run.ms Time spent executing all processing functions
executeplan.series-fetch.count Number of series fetched
executeplan.points-fetch.count Number of points fetched
executeplan.points-return.count Number of points returned
executeplan.cache-miss.count Number of cache misses (series with no useful chunks in cache)
executeplan.cache-hit-partial.count Number of partial cache hits (series with some useful chunks in cache)
executeplan.cache-hit.count Number of full cache hits (series with all needed chunks in cache)
executeplan.chunks-from-tank.count Number of chunks loaded from tank
executeplan.chunks-from-cache.count Number of chunks loaded from chunk cache
executeplan.chunks-from-store.count Number of chunks loaded from data storage
Series-specific lineage information

Every output series comes with lineage information. The lineage information is one or more lineage sections. Each section describes a certain lineage, and the number of input series matching that lineage that were part of the corresponding output series. Each lineage section has these fields:

Key Description
schema-name Name of the section in storage-schemas.conf
schema-retentions Retentions defined in storage-schemas.conf
archive-read Which archive was read as defined in the retentions. (0 means raw, 1 first rollup, etc)
archive-interval The native interval of the archive that was read
aggnum-norm If >1, number of points aggregated together per point, as part of normalization
aggnum-rc If >1, number of points aggregated together per output point, as part of runtime consolidation (MaxDataPoints)
consolidator-normfetch Consolidator used for normalization (if aggnum-norm > 1) and which rollup was read (if archive-read > 0)
consolidator-rc Consolidator used for runtime consolidation (MaxDataPoints) (if aggnum-rc > 1)
count Number of input series matching this lineage that were part of this output series