Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentationbreadcrumb arrow Grafana Enterprise Metrics Documentationbreadcrumb arrow Admin API
Enterprise

Admin API documentation

In addition to the standard Cortex API endpoints, Grafana Enterprise Metrics supports an admin HTTP API for managing cluster resources such as instances and tokens.

API operations

The API supports standard CRUD (create, read, update, and delete) operations for most resources. Creates are issued via POST requests, reads via GET requests, updates via PUT requests, and deletes via DELETE requests. When running multiple instances of the Admin API, we recommend that you enable leader election so that all mutations occur on a single, leader-elected instance.

Write consistency (If-Match and ETag headers)

To ensure concurrent changes are propagated in a way that avoids unintentional overwrites, an If-Match header is required for all requests that mutate data. The current version of a resource to be sent in the If-Match header can be found in the ETag header returned on all GET and PUT requests that return a single resource in the response body.

Here are a few examples of how to get the value of the ETag header using curl. The first example uses the argument -D - to write all headers to standard out, and then parses for the ETag header.

console 
curl -u :$API_TOKEN -sD - -o /dev/null http://localhost:8080/admin/api/v1/accesspolicies/{name} | grep -i ETag | cut -d " " -f 2

This second example uses the -i option to print the headers along with the response payload.

console 
curl -u :$API_TOKEN -i http://localhost:8080/admin/api/v1/accesspolicies/{name}

When making a request to mutate data, make sure to include the double quotes (") around the value in the If-Match header. So, for example, the following is a request to delete the access policy named the-access-policy, currently at version 1:

console 
curl -u :$API_TOKEN -X DELETE -H 'If-Match: "1"' http://localhost:8080/admin/api/v1/accesspolicies/the-access-policy

Optionally, a wildcard "*" may be passed as the If-Match version which effectively disables the write consistency protections and allows any current version to be updated.

Cluster API

List clusters

GET /admin/api/v1/clusters

Response:

json 
{
  "items": [
    {
      "name":  "<string>",              // cluster name
      "display_name": "<string>",       // cluster display name
      "created_at": "<rfc3339 string>", // date created 
      "kind": "<string>",               // cluster type, currently only cortex is supported
      "base_url": "<string>"            // Base URL of the cluster, optional
    }
    ...
  ],
  "type": "cluster"
}

Example:

console 
$ curl -u :$API_TOKEN http://localhost:8080/admin/api/v1/clusters | jq
{
  "items": [
    {
      "name": "metrics-enterprise-dev",
      "display_name": "metrics-enterprise-dev",
      "created_at": "2020-07-13T16:50:41.953793Z",
      "kind": "cortex",
      "base_url": ""
    }
  ],
  "type": "cluster"
}

Get cluster

GET /admin/api/v1/clusters/{name}

NOTES:

  • There is no ETag header on a cluster response because clusters are immutable.

Response:

json 
{
  "name":  "<string>",              // cluster name
  "display_name": "<string>",       // cluster display name
  "created_at": "<rfc3339 string>", // date created 
  "kind": "<string>",               // cluster type, currently only cortex is supported
  "base_url": "<string>"            // Base URL of the cluster, optional
}

Example:

console 
$ curl -u :$API_TOKEN http://localhost:8080/admin/api/v1/clusters/metrics-enterprise-dev | jq
{
  "name": "metrics-enterprise-dev",
  "display_name": "metrics-enterprise-dev",
  "created_at": "2020-07-13T16:50:41.953793Z",
  "kind": "cortex",
  "base_url": ""
}

Instance API

Model

The instance model is made up of the following fields:

  • name: The machine-readable name of an instance. It must be between 3 and 64 characters and only include the following characters, [a-z0-9-_]. Once set, this field is immutable.
  • display_name: The human-readable name of an instance. It can contain any set of characters and can be changed.
  • status: The current status of this instance. It can have the following values:
    • active
    • inactive
    • unknown
  • cluster: The name of the Grafana Enterprise Metrics cluster that this instance is scoped to. You will only be able to write/query this instance from this cluster.
  • limits: Instance specific resource usage limits. Limits are key-value pairs (JSON object) that correspond to Cortex limits configuration. This field is optional and will be omitted from responses when unset.

List instances

GET /admin/api/v1/instances

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_at": "<rfc3339 string>",
      "status": "<string>",
      "cluster": "<string>",
      "limits": {
        ...
      }
    },
    ...
  ],
  "type": "instance"
}

Example:

console 
$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances | jq
{
  "items": [
    {
      "name": "metrics-enterprise-dev",
      "display_name": "Grafana Enterprise Metrics Dev Instance",
      "created_at": "2020-07-13T17:37:59.341728283Z",
      "status": "active",
      "cluster": "metrics-enterprise-dev",
      "limits": {
        "ingestion_rate": 350000,
        "ingestion_rate_strategy": "local",
        "ingestion_burst_size": 350000,
        "accept_ha_samples": false,
        "ha_cluster_label": "cluster",
        "ha_replica_label": "__replica__",
        "ha_max_clusters": 0,
        "drop_labels": null,
        "max_label_name_length": 1024,
        "max_label_value_length": 2048,
        "max_label_names_per_series": 30,
        "max_metadata_length": 1024,
        "reject_old_samples": false,
        "reject_old_samples_max_age": 1209600000000000,
        "creation_grace_period": 600000000000,
        "enforce_metadata_metric_name": true,
        "enforce_metric_name": true,
        "ingestion_tenant_shard_size": 0,
        "max_series_per_query": 10002,
        "max_samples_per_query": 100000,
        "max_series_per_user": 10002,
        "max_series_per_metric": 0,
        "max_global_series_per_user": 300000,
        "max_global_series_per_metric": 300000,
        "min_chunk_length": 0,
        "max_metadata_per_user": 8000,
        "max_metadata_per_metric": 10,
        "max_global_metadata_per_user": 0,
        "max_global_metadata_per_metric": 0,
        "max_chunks_per_query": 2000000,
        "max_query_lookback": "0s",
        "max_query_length": 0,
        "max_query_parallelism": 14,
        "cardinality_limit": 100000,
        "max_cache_freshness": 60000000000,
        "max_queriers_per_tenant": 0,
        "ruler_evaluation_delay_duration": 0,
        "ruler_tenant_shard_size": 0,
        "ruler_max_rules_per_rule_group": 0,
        "ruler_max_rule_groups_per_tenant": 0,
        "store_gateway_tenant_shard_size": 0,
        "compactor_blocks_retention_period": 0,
        "s3_sse_type": "",
        "s3_sse_kms_key_id": "",
        "s3_sse_kms_encryption_context": "",
        "per_tenant_override_config": "",
        "per_tenant_override_period": 10000000000
      }
    }
  ],
  "type": "instance"
}

Create instance

POST /admin/api/v1/instances

Example:

console 
curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances \
--data '{"name":"metrics-enterprise-dev", "display_name":"Grafana Enterprise Metrics Dev Instance", "cluster": "metrics-enterprise-dev"}'

Update instance

PUT /admin/api/v1/instances/{name}

NOTES:

  • The name field of the object in the request body is ignored during updates.
  • The created_at field may not be modified during updates.
  • If-Match header matching the current version is required.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "status": "<string>",
  "cluster": "<string>",
  "limits": {
    ...
  }
}

Example:

console 
curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/metrics-enterprise-dev \
-X PUT -H 'If-Match: "123"` \
--data '{"display_name":"Grafana Enterprise Metrics Dev Instance", "cluster": "metrics-enterprise-dev"}' | jq
{
  "name": "metrics-enterprise-dev",
  "display_name": "Grafana Enterprise Metrics Dev Instance",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "metrics-enterprise-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_rate_strategy": "local",
    "ingestion_burst_size": 350000,
    "accept_ha_samples": false,
    "ha_cluster_label": "cluster",
    "ha_replica_label": "__replica__",
    "ha_max_clusters": 0,
    "drop_labels": null,
    "max_label_name_length": 1024,
    "max_label_value_length": 2048,
    "max_label_names_per_series": 30,
    "max_metadata_length": 1024,
    "reject_old_samples": false,
    "reject_old_samples_max_age": 1209600000000000,
    "creation_grace_period": 600000000000,
    "enforce_metadata_metric_name": true,
    "enforce_metric_name": true,
    "ingestion_tenant_shard_size": 0,
    "max_series_per_query": 10002,
    "max_samples_per_query": 100000,
    "max_series_per_user": 10002,
    "max_series_per_metric": 0,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "min_chunk_length": 0,
    "max_metadata_per_user": 8000,
    "max_metadata_per_metric": 10,
    "max_global_metadata_per_user": 0,
    "max_global_metadata_per_metric": 0,
    "max_chunks_per_query": 2000000,
    "max_query_lookback": "0s",
    "max_query_length": 0,
    "max_query_parallelism": 14,
    "cardinality_limit": 100000,
    "max_cache_freshness": 60000000000,
    "max_queriers_per_tenant": 0,
    "ruler_evaluation_delay_duration": 0,
    "ruler_tenant_shard_size": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0,
    "store_gateway_tenant_shard_size": 0,
    "compactor_blocks_retention_period": 0,
    "s3_sse_type": "",
    "s3_sse_kms_key_id": "",
    "s3_sse_kms_encryption_context": "",
    "per_tenant_override_config": "",
    "per_tenant_override_period": 10000000000
  }
}

Get instance

GET /admin/api/v1/instances/{name}

NOTES:

  • The current version of the instance will be returned in an ETag header on the response.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "status": "<string>",
  "cluster": "<string>",
  "limits": {
    ...
  }
}

Example:

console 
$ curl -u :$API_TOKEN localhost:8080/admin/api/v1/instances/metrics-enterprise-dev | jq
{
  "name": "metrics-enterprise-dev",
  "display_name": "Grafana Enterprise Metrics Dev Instance",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "metrics-enterprise-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_rate_strategy": "local",
    "ingestion_burst_size": 350000,
    "accept_ha_samples": false,
    "ha_cluster_label": "cluster",
    "ha_replica_label": "__replica__",
    "ha_max_clusters": 0,
    "drop_labels": null,
    "max_label_name_length": 1024,
    "max_label_value_length": 2048,
    "max_label_names_per_series": 30,
    "max_metadata_length": 1024,
    "reject_old_samples": false,
    "reject_old_samples_max_age": 1209600000000000,
    "creation_grace_period": 600000000000,
    "enforce_metadata_metric_name": true,
    "enforce_metric_name": true,
    "ingestion_tenant_shard_size": 0,
    "max_series_per_query": 10002,
    "max_samples_per_query": 100000,
    "max_series_per_user": 10002,
    "max_series_per_metric": 0,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "min_chunk_length": 0,
    "max_metadata_per_user": 8000,
    "max_metadata_per_metric": 10,
    "max_global_metadata_per_user": 0,
    "max_global_metadata_per_metric": 0,
    "max_chunks_per_query": 2000000,
    "max_query_lookback": "0s",
    "max_query_length": 0,
    "max_query_parallelism": 14,
    "cardinality_limit": 100000,
    "max_cache_freshness": 60000000000,
    "max_queriers_per_tenant": 0,
    "ruler_evaluation_delay_duration": 0,
    "ruler_tenant_shard_size": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0,
    "store_gateway_tenant_shard_size": 0,
    "compactor_blocks_retention_period": 0,
    "s3_sse_type": "",
    "s3_sse_kms_key_id": "",
    "s3_sse_kms_encryption_context": "",
    "per_tenant_override_config": "",
    "per_tenant_override_period": 10000000000
  }
}

Delete instance

DELETE /admin/api/v1/instances/{name}

NOTES:

  • If-Match header matching the current version is required.

Example:

console 
curl -X "DELETE" -u :$API_TOKEN -H 'If-Match: "123"' localhost:8080/admin/api/v1/instances/metrics-enterprise-dev

Access policy API

List access policies

GET /admin/api/v1/accesspolicies

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_at": "<rfc3339 string>",
      "realms": [
        {
          "instance": "<string>",
          "cluster": "<string>"
        }
      ],
      "expiration": "<rfc3339 string>",
      "scopes": [
        "<string>",
        ...
      ]
    },
    ...
  ],
  "type": "key"
}

Create access policy

POST /admin/api/v1/accesspolicies

Payload:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "realms": [
    {
      "instance": "<string>",
      "cluster": "<string>"
    }
  ],
  "expiration": "<rfc3339 string>",
  "scopes": [
    "<string>",
    ...
  ]
}

Realms

Realms designate the instance/cluster pairs that the access policy allows requests for:

  • instance: This field must be set to an existing instance or *. * denotes access to all instances.
  • cluster: This field must be set to an existing cluster.
  • label_policies: This is an optional field that can be set to a list of label policies. An access policy gets access to metrics that match at least one of the label policies. If label policies are not set, all metrics will match.
Label policy, selector, and label matchers

A label policy contains a selector, that consists of a list of label matchers.

The following label matcher types are supported:

  • EQ: Select labels that are exactly equal to the provided string.
  • NEQ: Select labels that are not equal to the provided string.
  • RE: Select labels that regex-match the provided string.
  • NRE: Select labels that do not regex-match the provided string.
json 
{
  "selector": [
    {
      "type": "<enum: EQ | NEQ | RE | NRE>",
      "name": "<string>",
      "value": "<string>"
    }
  ]
}

The PromQL selector {job="grafana-labs", role!="software-engineer"}, would translate into this label policy:

json 
{
  "selector": [
    {
      "type": "EQ",
      "name": "job",
      "value": "grafana-labs"
    },
    {
      "type": "NE",
      "name": "role",
      "value": "software-engineer"
    }
  ]
}

Scopes

Scopes designate what operations tokens assigned to this access policy will be able to do when calling the GEM API.

  • metrics:read: Permission to view data from an instance
  • metrics:write: Permission to write data to an instance
  • metrics:delete: Permission to delete data from an instance
  • rules:read: Permission to read ruler rules
  • rules:write: Permission to write ruler rules
  • admin: Permission to perform admin operations

Update access policy

PUT /admin/api/v1/accesspolicies/{name}

NOTES:

  • If-Match header matching the current version is required.
  • Update differs from create in that only the Display Name, Realms, and Scopes are updatable.

Payload:

json 
{
  "display_name": "<string>",
  "realms": [
    {
      "instance": "<string>",
      "cluster": "<string>"
    }
  ],
  "scopes": [
    "<string>",
    ...
  ]
}

Get access policy

GET /admin/api/v1/accesspolicies/{name}

NOTES:

  • The current version of the access policy will be returned in an ETag header on the response.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "realms": [
    {
      "instance": "<string>",
      "cluster": "<string>"
    }
  ],
  "expiration": "<rfc3339 string>",
  "scopes": [
    "<string>",
    ...
  ]
}

Delete access policy

DELETE /admin/api/v1/accesspolicies/{name}

NOTES:

  • If-Match header matching the current version is required.

Example:

console 
$ curl -X "DELETE" -u :$API_TOKEN -H 'If-Match: "123"' localhost:8080/admin/api/v1/accesspolicies/dev-read-write

Token API

List tokens

GET /admin/api/v1/tokens

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_at": "<rfc3339 string>",
      "expiration": "<rfc3339 string>",
      "access_policy": "<string>"
    },
    ...
  ],
  "type": "key"
}

Create token

POST /admin/api/v1/tokens

Payload:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "expiration": "<rfc3339 string>",
  "access_policy": "<string>"
}

Get token

GET /admin/api/v1/tokens/{name}

NOTES:

  • The current version of the token will be returned in an ETag header on the response.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "expiration": "<rfc3339 string>",
  "access_policy": "<string>"
}

Delete token

DELETE /admin/api/v1/tokens/{name}

NOTES:

  • If-Match header matching the current version is required.

Example:

console 
curl -X "DELETE" -u :$API_TOKEN -H 'If-Match: "123"' localhost:8080/admin/api/v1/token/dev-read-write

License API

List licenses

GET /admin/api/v1/licenses

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_at": "<rfc3339 string>",
      "token": {
        "jti": "<string>",
        "iss": "<string>",
        "sub": "<string>",
        "iat": "<int64>",
        "exp": "<int64>",
        "nbf": "<int64>",
        "lexp": "<int64>",
        "lid": "<string>",
        "max_users": "<int64>",
        "included_admins": "<int64>",
        "included_viewers": "<int64>",
        "lic_exp_warn_days": "<int64>",
        "prod": [
          "<string>",
          ...
        ],
        "company": "<string>",
        "slug": "<string>",
      }
    },
    ...
  ],
  "type": "license"
}

Feature detection API

Get product and feature information

GET /admin/api/v1/features

Response:

json 
{
  "product": "<string>",
  "product_version": "<string>",
  "features": {
      "<string>": "<string>",
      ...
  }
}

Example response:

json 
{
  "product": "GEM",
  "product_version": "1.1",
  "features": {
      "lbac": "v2",
      "self_monitoring": "v1"
  }
}