Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Enterprise Metrics documentation Admin API
Enterprise

Admin API documentation

NOTE: Admin API version v1 /admin/api/v1/ and v2 /admin/api/v2/ endpoints are deprecated and will be removed in a future release of GEM. Use the new Admin API version v3 endpoints /admin/api/v3/ instead.

In addition to the standard Grafana Mimir API endpoints, Grafana Enterprise Metrics supports an admin HTTP API for managing cluster resources such as tenants and tokens.

API operations

The API supports standard CRUD (create, read, update, and delete) operations for most resources. Creates are issued via POST requests, reads via GET requests, updates via PUT requests, and deletes via DELETE requests. When running multiple instances of the Admin API, we recommend that you enable leader election so that all mutations occur on a single, leader-elected instance.

Write consistency (If-Match and ETag headers)

To ensure concurrent changes are propagated in a way that avoids unintentional overwrites, an If-Match header is required for all requests that mutate data. The current version of a resource to be sent in the If-Match header can be found in the ETag header returned on all GET and PUT requests that return a single resource in the response body.

Here are a few examples of how to get the value of the ETag header using curl. The first example uses the argument -D - to write all headers to standard out, and then parses for the ETag header.

console 
curl -u :$API_TOKEN -sD - -o /dev/null http://localhost:8080/admin/api/v3/accesspolicies/{name} | grep -i ETag | cut -d " " -f 2

This second example uses the -i option to print the headers along with the response payload.

console 
curl -u :$API_TOKEN -i http://localhost:8080/admin/api/v3/accesspolicies/{name}

When making a request to mutate data, make sure to include the double quotes (") around the value in the If-Match header. So, for example, the following is a request to update the access policy named the-access-policy, currently at version 1:

console 
curl -u :$API_TOKEN -X PUT -H 'If-Match: "1"' http://localhost:8080/admin/api/v3/accesspolicies/the-access-policy --data '{"status": "active", "realms": [{"tenant": "enterprise-metrics-dev", "cluster": "enterprise-metrics-dev"}], "scopes": ["metrics:write"]}'

Optionally, a wildcard "*" may be passed as the If-Match version which effectively disables the write consistency protections and allows any current version to be updated.

Cluster API

List clusters

GET /admin/api/v1/clusters GET /admin/api/v2/clusters

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/clusters instead.

GET /admin/api/v3/clusters

Response:

json 
{
  "items": [
    {
      "name":  "<string>",              // cluster name
      "display_name": "<string>",       // cluster display name
      "created_at": "<rfc3339 string>", // date created
      "kind": "<string>",               // cluster type, currently only cortex is supported
      "base_url": "<string>"            // Base URL of the cluster, optional
    }
    ...
  ],
  "type": "cluster"
}

Example:

console 
$ curl -u :$API_TOKEN http://localhost:8080/admin/api/v3/clusters | jq
{
  "items": [
    {
      "name": "enterprise-metrics-dev",
      "display_name": "enterprise-metrics-dev",
      "created_at": "2020-07-13T16:50:41.953793Z",
      "kind": "metrics",
      "base_url": ""
    }
  ],
  "type": "cluster"
}

Get cluster

GET /admin/api/v1/clusters/{name} GET /admin/api/v2/clusters/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/clusters/{name} instead.

GET /admin/api/v3/clusters/{name}

NOTES:

  • There is no ETag header on a cluster response because clusters are immutable.

Response:

json 
{
  "name": "<string>", // cluster name
  "display_name": "<string>", // cluster display name
  "created_at": "<rfc3339 string>", // date created
  "kind": "<string>", // cluster type, currently only cortex is supported
  "base_url": "<string>" // Base URL of the cluster, optional
}

Example:

console 
$ curl -u :$API_TOKEN http://localhost:8080/admin/api/v3/clusters/enterprise-metrics-dev | jq
{
  "name": "enterprise-metrics-dev",
  "display_name": "enterprise-metrics-dev",
  "created_at": "2020-07-13T16:50:41.953793Z",
  "kind": "metrics",
  "base_url": ""
}

Tenant API

Model

The tenant model is made up of the following fields:

  • name: The machine-readable name of a tenant. It must be between 3 and 64 characters and only include the following characters, [a-z0-9-_]. Once set, this field is immutable.
  • display_name: The human-readable name of a tenant. It can contain any set of characters and can be changed.
  • status: The current status of this tenant. It can have the following values:
    • active
    • inactive
    • unknown
  • cluster: The name of the Grafana Enterprise Metrics cluster that this tenant is scoped to. You will only be able to write/query this tenant from this cluster.
  • limits: Tenant specific resource usage limits. Limits are key-value pairs (JSON object) that correspond to Grafana Mimir limits configuration. This field is optional and will be omitted from responses when unset.

List tenants

GET /admin/api/v1/instances GET /admin/api/v2/tenants

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/tenants instead.

GET /admin/api/v3/tenants

NOTES:

  • This endpoint only returns tenants in an active status. To get all tenants including inactive ones use the query parameter include-non-active=true

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_at": "<rfc3339 string>",
      "status": "<string>",
      "cluster": "<string>",
      "limits": {
        ...
      }
    },
    ...
  ],
  "type": "tenant"
}

Example:

console 
$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants | jq
{
  "items": [
    {
      "name": "enterprise-metrics-dev",
      "display_name": "Grafana Enterprise Metrics dev tenant",
      "created_at": "2020-07-13T17:37:59.341728283Z",
      "status": "active",
      "cluster": "enterprise-metrics-dev",
      "limits": {
        "ingestion_rate": 350000,
        "ingestion_burst_size": 350000,
        "max_series_per_query": 100000,
        "max_global_series_per_user": 300000,
        "max_global_series_per_metric": 300000,
        "ruler_max_rules_per_rule_group": 0,
        "ruler_max_rule_groups_per_tenant": 0
      }
    }
  ],
  "type": "tenant"
}

Create tenant

POST /admin/api/v1/instances POST /admin/api/v2/tenants

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint POST /admin/api/v3/tenants instead.

POST /admin/api/v3/tenants

NOTES:

  • Because tenant names are unique, you cannot create a new tenant with the same name as a tenant that already exists but is in the inactive state. Instead, to use the same tenant name again you must re-enable the inactive tenant by updating its state to active. To make this change, use the Update Tenant endpoint. When creating a tenant via this API call, it is not required to specify the status field because the status is always overwritten by the API to active.
  • To manage the limits of the tenant via the runtime configuration set limits: null or omit the limits field from the request. GEM will use the default values of the limits and will ignore the runtime configuration when limits: {}.

Example:

$ curl -u :$API_TOKEN -X POST localhost:8080/admin/api/v3/tenants --data-raw '
{
  "name": "enterprise-metrics-dev",
  "display_name": "Grafana Enterprise Metrics dev tenant",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "ruler_max_rule_groups_per_tenant": 1
  }
}'

Response:

json 
{
  "name": "enterprise-metrics-dev",
  "display_name": "Grafana Enterprise Metrics dev tenant",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_burst_size": 350000,
    "max_series_per_query": 100000,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 1,
    ...
  }
}

Update tenant

PUT /admin/api/v1/instances/{name} PUT /admin/api/v2/tenants/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint PUT /admin/api/v3/tenants/{name} instead.

PUT /admin/api/v3/tenants/{name}

NOTES:

  • The name field of the object in the request body is ignored during updates.
  • The created_at field may not be modified during updates.
  • If-Match header matching the current version is required.
  • To manage the limits of the tenant via the runtime configuration set limits: null or omit the limits field from the request. GEM will use the default values of the limits and will ignore the runtime configuration when limits: {}.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "status": "<string>",
  "cluster": "<string>",
  "limits": {
    ...
  }
}

Example:

console 
curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/enterprise-metrics-dev \
-X PUT -H 'If-Match: "123"` \
--data '{"display_name":"Grafana Enterprise Metrics dev tenant", "status": "active", "cluster": "enterprise-metrics-dev"}' | jq
{
  "name": "enterprise-metrics-dev",
  "display_name": "Grafana Enterprise Metrics dev tenant",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_burst_size": 350000,
    "max_series_per_query": 100000,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "max_global_exemplars_per_user": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0
  }
}

Get tenant

GET /admin/api/v1/instances/{name} GET /admin/api/v2/tenants/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/tenants/{name} instead.

GET /admin/api/v3/tenants/{name}

NOTES:

  • The current version of the tenant will be returned in an ETag header on the response.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "status": "<string>",
  "cluster": "<string>",
  "limits": {
    ...
  }
}

Example:

console 
$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/enterprise-metrics-dev | jq
{
  "name": "enterprise-metrics-dev",
  "display_name": "Grafana Enterprise Metrics dev tenant",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_burst_size": 350000,
    "max_series_per_query": 100000,
    "max_global_series_per_user": 300000,
    "max_global_series_per_metric": 300000,
    "max_global_exemplars_per_user": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0
  }
}

Delete tenant

DELETE /admin/api/v1/instances/{name} DELETE /admin/api/v2/instances/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. In Admin API version v3, delete operations are no longer supported and have been replaced by soft deletes. Use the endpoint PUT /admin/api/v3/tenants/{name} with "status": "inactive" instead. This operation will inactivate a tenant and subsequent HTTP requests for the tenant will fail with a 401 Unauthorized HTTP status code. To re-enable a tenant, use the endpoint PUT /admin/api/v3/tenants/{name} with "status": "active".

PUT /admin/api/v3/tenants/{name}

json 
{
  "status": "inactive",
  "cluster": "<string>"
}

NOTES:

  • If-Match header matching the current version is required.

Example:

console 
curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/enterprise-metrics-dev \
-X PUT -H 'If-Match: "123"` \
--data '{"status": "inactive", "cluster": "enterprise-metrics-dev"}' | jq
{
  "name": "enterprise-metrics-dev",
  "display_name": "Grafana Enterprise Metrics dev tenant",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "inactive",
  "cluster": "enterprise-metrics-dev"
}

Access policy API

List access policies

GET /admin/api/v1/accesspolicies GET /admin/api/v2/accesspolicies

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/accesspolicies instead.

GET /admin/api/v3/accesspolicies

NOTES:

  • This endpoint only returns access policies in an active status. To get all access policies including inactive ones use the query parameter include-non-active=true

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_at": "<rfc3339 string>",
      "status": "<string>",
      "realms": [
        {
          "tenant": "<string>",
          "cluster": "<string>"
        }
      ],
      "scopes": [
        "<string>",
        ...
      ]
    },
    ...
  ],
  "type": "access_policy"
}

Create access policy

POST /admin/api/v1/accesspolicies POST /admin/api/v2/accesspolicies

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint POST /admin/api/v3/accesspolicies instead.

POST /admin/api/v3/accesspolicies

Payload:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "realms": [
    {
      "tenant": "<string>",
      "cluster": "<string>"
    }
  ],
  "scopes": [
    "<string>",
    ...
  ]
}

Because access policy names are unique, you cannot create a new access policy with the same name as an access policy that already exists but is in the inactive state. Instead, to use the same access policy name again you must re-enable the inactive access policy by updating its state to active. To make this change, use the Update Access Policy endpoint. When creating an access policy via this API call, it is not required to specify the status field because the status is always overwritten by the API to active.

Realms

Realms designate the tenant/cluster pairs that the access policy allows requests for:

  • tenant: This field must be set to an existing tenant or *. * denotes access to all tenants.
  • cluster: This field must be set to an existing cluster.
  • label_policies: This is an optional field that can be set to a list of label policies. An access policy gets access to metrics that match at least one of the label policies. If label policies are not set, all metrics will match.
Label policy, selector, and label matchers

A label policy contains a selector, that consists of a list of label matchers.

The following label matcher types are supported:

  • EQ: Select labels that are exactly equal to the provided string.
  • NEQ: Select labels that are not equal to the provided string.
  • RE: Select labels that regex-match the provided string.
  • NRE: Select labels that do not regex-match the provided string.
json 
{
  "selector": [
    {
      "type": "<enum: EQ | NEQ | RE | NRE>",
      "name": "<string>",
      "value": "<string>"
    }
  ]
}

The PromQL selector {job="grafana-labs", role!="software-engineer"}, would translate into this label policy:

json 
{
  "selector": [
    {
      "type": "EQ",
      "name": "job",
      "value": "grafana-labs"
    },
    {
      "type": "NE",
      "name": "role",
      "value": "software-engineer"
    }
  ]
}

Scopes

Scopes designate what operations tokens assigned to this access policy will be able to do when calling the GEM API.

  • admin: Permission to perform admin operations
  • admin:read: Permission to view admin resources (access policies and tokens, tenants, and clusters)
  • alerts:read: Permission to view the Alertmanager UI and get the Alertmanager configuration for a tenant
  • alerts:write: Permission to create silences in the Alertmanager UI and create and delete the Alertmanager configuration for a tenant
  • metrics:delete: Permission to delete data from a tenant
  • metrics:read: Permission to view data from a tenant
  • metrics:write: Permission to write data to a tenant
  • rules:read: Permission to read ruler rules
  • rules:write: Permission to write ruler rules

Update access policy

PUT /admin/api/v1/accesspolicies/{name} PUT /admin/api/v2/accesspolicies/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint PUT /admin/api/v3/accesspolicies/{name} instead.

PUT /admin/api/v3/accesspolicies/{name}

NOTES:

  • If-Match header matching the current version is required.
  • Update differs from create in that only the Display Name, Status, Realms, and Scopes are updatable.

Payload:

json 
{
  "display_name": "<string>",
  "status": "<string>",
  "realms": [
    {
      "tenant": "<string>",
      "cluster": "<string>"
    }
  ],
  "scopes": [
    "<string>",
    ...
  ]
}

Get access policy

GET /admin/api/v1/accesspolicies/{name} GET /admin/api/v2/accesspolicies/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/accesspolicies/{name} instead.

GET /admin/api/v3/accesspolicies/{name}

NOTES:

  • The current version of the access policy will be returned in an ETag header on the response.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "status": "<string>",
  "realms": [
    {
      "tenant": "<string>",
      "cluster": "<string>"
    }
  ],
  "scopes": [
    "<string>",
    ...
  ]
}

Delete access policy

DELETE /admin/api/v1/accesspolicies/{name} DELETE /admin/api/v2/accesspolicies/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. In Admin API version v3, delete operations are no longer supported and have been replaced by soft deletes. Use the endpoint PUT /admin/api/v3/accesspolicies/{name} with "status": "inactive" instead. This operation will inactivate an access policy and subsequent HTTP requests with an associated token will fail with a 401 Unauthorized HTTP status code. To re-enable an access policy, use the endpoint PUT /admin/api/v3/accesspolicies/{name} with "status": "active".

PUT /admin/api/v3/accesspolicies/{name}

NOTES:

  • If-Match header matching the current version is required.

Example:

console 
curl -u :$API_TOKEN localhost:8080/admin/api/v3/tokens/accesspolicies/enterprise-metrics-ap \
-X PUT -H 'If-Match: "123"` \
--data '{"status": "inactive", "realms": [{"tenant": "enterprise-metrics-dev", "cluster": "enterprise-metrics-dev"}], "scopes": ["metrics:write"]}' | jq
{
  "name": "enterprise-metrics-ap",
  "created_at": "2022-02-11T16:55:43.598543386Z",
  "status": "inactive",
  "realms": [
      {
          "tenant": "enterprise-metrics-dev",
          "cluster": "enterprise-metrics-dev"
      }
  ],
  "scopes": [
      "metrics:write"
  ]
}

Token API

List tokens

GET /admin/api/v1/tokens GET /admin/api/v2/tokens

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/tokens instead.

GET /admin/api/v3/tokens

NOTES:

  • This endpoint only returns tokens in an active status. To get all tokens including inactive ones use the query parameter include-non-active=true

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_by": "<string>",
      "created_at": "<rfc3339 string>",
      "status": "<string>",
      "access_policy": "<string>",
      "expiration": "<rfc3339 string>"
    },
    ...
  ],
  "type": "token"
}

Create token

POST /admin/api/v1/tokens POST /admin/api/v2/tokens

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint POST /admin/api/v3/tokens instead.

POST /admin/api/v3/tokens

Payload:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "access_policy": "<string>",
  "expiration": "<rfc3339 string>"
}

Because token names are unique, you cannot create a new token with the same name as a token that already exists but is in the inactive state. Instead, to use the same token name again you must re-enable the inactive token by updating its state to active. To make this change, use the PUT /admin/api/v3/tokens/{name} endpoint. When creating a token via this API call, it is not required to specify the status field because the status is always overwritten by the API to active.

Get token

GET /admin/api/v1/tokens/{name} GET /admin/api/v2/tokens/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/tokens/{name} instead.

GET /admin/api/v3/tokens/{name}

NOTES:

  • The current version of the token will be returned in an ETag header on the response.

Response:

json 
{
  "name": "<string>",
  "display_name": "<string>",
  "created_at": "<rfc3339 string>",
  "status": "<string>",
  "access_policy": "<string>",
  "expiration": "<rfc3339 string>"
}

Delete token

DELETE /admin/api/v1/tokens/{name} DELETE /admin/api/v2/tokens/{name}

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. In Admin API version v3, delete operations are no longer supported and have been replaced by soft deletes. Use the endpoint PUT /admin/api/v3/tokens/{name} with "status": "inactive" instead. This operation will inactivate a token and subsequent HTTP requests with the token will fail with a 401 Unauthorized HTTP status code. To re-enable a token, use the endpoint PUT /admin/api/v3/tokens/{name} with "status": "active".

PUT /admin/api/v3/tokens/{name}

json 
{
  "status": "inactive"
}

NOTES:

  • If-Match header matching the current version is required.

Example:

console 
curl -u :$API_TOKEN localhost:8080/admin/api/v3/tokens/enterprise-metrics-token \
-X PUT -H 'If-Match: "123"` \
--data '{"status": "inactive"}' | jq
{
  "name": "enterprise-metrics-token",
  "display_name": "Grafana Enterprise Metrics Dev Token",
  "created_at": "2022-02-11T16:59:36.275826382Z",
  "status": "inactive",
  "access_policy": "enterprise-metrics-ap",
  "expiration": "2050-01-01T00:00:00Z"
}

License API

List licenses

GET /admin/api/v1/licenses GET /admin/api/v2/licenses

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/licenses instead.

GET /admin/api/v3/licenses

Response:

json 
{
  "items": [
    {
      "name": "<string>",
      "display_name": "<string>",
      "created_by": "<string>",
      "created_at": "<rfc3339 string>",
      "token": {
        "jti": "<string>",
        "iss": "<string>",
        "sub": "<string>",
        "iat": "<int64>",
        "exp": "<int64>",
        "nbf": "<int64>",
        "lexp": "<int64>",
        "lid": "<string>",
        "max_users": "<int64>",
        "included_admins": "<int64>",
        "included_viewers": "<int64>",
        "lic_exp_warn_days": "<int64>",
        "prod": [
          "<string>",
          ...
        ],
        "company": "<string>",
        "slug": "<string>",
      }
    },
    ...
  ],
  "type": "license"
}

Feature detection API

Get product and feature information

GET /admin/api/v1/features GET /admin/api/v2/features

Deprecated: these endpoints are deprecated and will be removed in a future release of GEM. Use the endpoint GET /admin/api/v3/features instead.

GET /admin/api/v3/features

Response:

json 
{
  "name": "<string>",
  "version": "<string>",
  "features": {
      "<string>": "<string>",
      ...
  }
}

Example response:

json 
{
  "name": "GEM",
  "version": "1.1",
  "features": {
    "debug_export": "v1",
    "editable_access_policies": "v1",
    "editable_tenants": "v1",
    "lbac": "v1",
    "self_monitoring": "v1",
    "federated_rules": "v1"
  }
}