Documentation Grafana Mimir References HTTP API

Grafana Mimir HTTP API

Grafana Mimir exposes an HTTP API that you can use to write and query time series data, and operate the cluster.

This document groups API endpoints by service. Note that the API endpoints are exposed when you run Grafana Mimir in microservices mode, monolithic mode, and read-write mode:

• Microservices mode: Each service exposes its own endpoints.
• Monolithic mode: The Grafana Mimir instance exposes all API endpoints.
• Read-write mode: The component services are exposed on the endpoint that they are contained within. Either Mimir read, Mimir write, or Mimir backend. Refer to Deployment modes for the grouping of components.

Endpoints

Path prefixes

The following table provides usage of placeholder path prefixes, for prefixes that are configurable.

Authentication

Endpoints that require authentication must be called with the X-Scope-OrgID HTTP request header specified to the tenant ID.

If you disable multi-tenancy, Grafana Mimir doesn’t require any request to include the X-Scope-OrgID header.

Multi-tenancy can be enabled and disabled via the -auth.multitenancy-enabled flag or its respective YAML configuration option.

For more information about authentication and authorization, refer to Authentication and Authorization.

All services

The following API endpoints are exposed by all services.

Index page

Copy

GET /

This endpoint displays an index page with links to other web pages exposed by Grafana Mimir.

Configuration

Copy

GET /config

This endpoint displays the configuration currently applied to Grafana Mimir including default values and settings via CLI flags. This endpoint provides the configuration in YAML format and masks sensitive data.

Different modes

Copy

GET /config?mode=diff

This endpoint displays the differences between the Grafana Mimir default configuration and the current configuration.

Copy

GET /config?mode=defaults

This endpoint displays the default configuration values.

Status Configuration

Copy

GET /api/v1/status/config

This endpoint displays empty configuration settings, it exists only to be compatible with the Prometheus /api/v1/status/config API.

Status Flags

Copy

GET /api/v1/status/flags

This endpoint displays empty configuration flags, it exists only to be compatible with the Prometheus /api/v1/status/flags API.

Runtime Configuration

Copy

GET /runtime_config

This endpoint displays the runtime configuration currently applied to Grafana Mimir, in YAML format, including default values. The endpoint is only available if Grafana Mimir is configured with the -runtime-config.file option.

Different modes

Copy

GET /runtime_config?mode=diff

This endpoint displays the differences between the Grafana Mimir default runtime configuration and the current runtime configuration.

Services’ status

Copy

GET /services

This endpoint displays a web page with the status of internal Grafana Mimir services.

Readiness probe

Copy

GET /ready

This endoint returns 200 when Grafana Mimir is ready to serve traffic.

Metrics

Copy

GET /metrics

This endpoint returns the metrics for the running Grafana Mimir service in the Prometheus exposition format.

Pprof

Copy

GET /debug/pprof/heap
GET /debug/pprof/block
GET /debug/pprof/profile
GET /debug/pprof/trace
GET /debug/pprof/goroutine
GET /debug/pprof/mutex

These endpoints return runtime profiling data in the format expected by the pprof visualization tool. There are many things that can be profiled using this endpoint, including heap, trace, goroutine, and so on.

For more information about pprof, refer to pprof.

Fgprof

Copy

GET /debug/fgprof

This endpoint returns the sampling Go profiling data that you can use to analyze On-CPU and Off-CPU (for example, I/O) time.

For more information about fgprof, refer to fgprof.

Build information

Copy

GET /api/v1/status/buildinfo
GET <prometheus-http-prefix>/api/v1/status/buildinfo
GET <alertmanager-http-prefix>/api/v1/status/buildinfo

This endpoint returns in JSON format information about the build and enabled features. The format returned is not identical, but is similar to the Prometheus Build Information endpoint.

Format query

Copy

GET <prometheus-http-prefix>/api/v1/format_query?query={query}
POST <prometheus-http-prefix>/api/v1/format_query

Formats the PromQL query.

This endpoint is compatible with the Prometheus format query endpoint.

For more information about formatting queries, refer to Prometheus’ documentation.

Memberlist cluster

Copy

GET /memberlist

This admin page shows information about Memberlist cluster (list of nodes and their health) and KV store (keys and values in the KV store).

If memberlist message history is enabled, this page also shows all received and sent messages stored in the buffers. This can be useful for troubleshooting memberlist cluster. To enable message history buffers use -memberlist.message-history-buffer-bytes CLI flag or the corresponding YAML configuration parameter.

Get tenant limits

Copy

GET /api/v1/user_limits

Returns realtime limits for the authenticated tenant, in JSON format. This API is experimental.

Requires authentication.

The endpoint is only available if Grafana Mimir is configured with the -runtime-config.file option.

Distributor

The following endpoints relate to the distributor.

Remote write

Copy

POST /api/v1/push

Entrypoint for the Prometheus remote write.

This endpoint accepts an HTTP POST request with a body that contains a request encoded with Protocol Buffers and compressed with Snappy. You can find the definition of the protobuf message in pkg/mimirpb/mimir.proto. The HTTP request must contain the header X-Prometheus-Remote-Write-Version set to 0.1.0.

To skip the label name validation, perform the following actions:

• Enable API’s flag -api.skip-label-name-validation-header-enabled=true
• Ensure that the request is sent with the header X-Mimir-SkipLabelNameValidation: true

This feature supports the writes from non-standard downstream clients that have metric name not Prometheus compliant.

For more information, refer to Prometheus Remote storage integrations.

Requires authentication.

OTLP

Copy

POST /otlp/v1/metrics

Entrypoint for the OTLP HTTP.

This endpoint accepts an HTTP POST request with a body that contains a request encoded with Protocol Buffers and optionally compressed with GZIP. You can find the definition of the protobuf message in metrics.proto.

Requires authentication.

Influx

Copy

POST /api/v1/push/influx/write

Entry point for the Influx HTTP.

This endpoint accepts an HTTP POST request with a body that contains a request encoded in text/plain and is optionally compressed with GZIP. You can find more information on the protocol in InfluxDB line protocol.

This endpoint requires authentication.

Distributor ring status

Copy

GET /distributor/ring

This endpoint displays a web page with the distributor hash ring status, including the state, and the health and last heartbeat time of each distributor.

Tenants stats

Copy

GET /distributor/all_user_stats

This endpoint displays a web page that shows per-tenant statistics updated in real time, including the total number of active series across all ingesters and the current ingestion rate displayed in samples per second.

HA tracker status

Copy

GET /distributor/ha_tracker

This endpoint displays a web page with the current status of the HA tracker, including the elected replica for each Prometheus HA cluster.

Ingester

The following endpoints relate to the ingester.

Flush chunks / blocks

Copy

GET,POST /ingester/flush

This endpoint triggers a flush of the in-memory series time series data to the long-term storage. This endpoint also triggers the flush when -blocks-storage.tsdb.flush-blocks-on-shutdown is disabled.

This endpoint accepts a tenant parameter to specify the tenant whose blocks are compacted and shipped. This parameter might be specified multiple times to select more tenants. If no tenant is specified, all tenants are flushed.

The flush endpoint also accepts a wait=true parameter, which makes the call synchronous, and only returns a status code after flushing completes.

Prepare for Shutdown

Copy

GET,POST,DELETE /ingester/prepare-shutdown

This endpoint inspects or changes in-memory ingester configuration to prepare for permanently stopping an ingester instance but does not actually stop any part of the ingester.

After a POST to the prepare-shutdown endpoint returns, when the ingester process is stopped with SIGINT / SIGTERM, the ingester will be unregistered from the ring and in-memory time series data will be flushed to long-term storage. This endpoint causes the ingester to be unregistered from the ring when stopped even if you disable -ingester.ring.unregister-on-shutdown.

A GET to the prepare-shutdown endpoint returns the status of this configuration, either set or unset.

A DELETE to the prepare-shutdown endpoint reverts the configuration of the ingester to its previous state (with respect to unregistering on shutdown and flushing of in-memory time series data to long-term storage).

This API endpoint is usually used by Kubernetes-specific scale down automations such as the rollout-operator.

Shutdown

Copy

POST /ingester/shutdown

This endpoint flushes in-memory time series data from ingesters to the long-term storage, and then shuts down the ingester service. After the shutdown endpoint returns, the operator or any automation that’s used terminates the process with a SIGINT / SIGTERM signal. During this time, /ready does not return 200. This endpoint unregisters the ingester from the ring even if you disable -ingester.ring.unregister-on-shutdown.

This API endpoint is usually used by scale down automations.

Prepare partition downscale

Copy

GET,POST,DELETE /ingester/prepare-partition-downscale

This endpoint prepares the ingester’s partition for downscaling by setting it to the INACTIVE state.

A GET call to this endpoint returns a timestamp of when the partition was switched to the INACTIVE state, or 0, if the partition is not in the INACTIVE state.

A POST call switches this ingester’s partition to the INACTIVE state, if it isn’t INACTIVE already, and returns the timestamp of when the switch to the INACTIVE state occurred.

A DELETE call sets the partition back from the INACTIVE to the ACTIVE state.

If the ingester is not configured to use ingest-storage, any call to this endpoint fails.

This API endpoint is usually used by scale down automation, e.g. rollout-operator.

Prepare instance ring downscale

Copy

GET,POST,DELETE /ingester/prepare-instance-ring-downscale

This endpoint prepares the ingester for downscaling by setting it to read-only mode.

A GET call to this endpoint returns a timestamp of when the ingester was switched to read-only mode, or 0, if the ingester is not in read-only mode.

A POST call switches this ingester’s partition to read-only mode, if it isn’t read-only already, and returns the timestamp of when the switch to read-only mode occurred.

A DELETE call sets the ingester back to read-write mode.

If the ingester is configured to use ingest-storage, any call to this endpoint fails.

This API endpoint is usually used by scale down automation, e.g. rollout-operator.

Prepare for unregister

Copy

GET,PUT,DELETE /ingester/unregister-on-shutdown

This endpoint controls whether an ingester should unregister from the ring on its next termination, that is, the next time it receives a SIGINT or SIGTERM signal. Via this endpoint, Mimir operators can dynamically control an ingester’s -ingester.ring.unregister-on-shutdown state without having to restart the ingester.

A PUT sets the ingester’s unregister state. When invoked with the PUT method, the endpoint takes a request body:

Copy

{"unregister": true}

A GET returns the ingester’s current unregister state.

A DELETE resets the ingester’s unregister state to the value that was passed via the -ingester.ring.unregister-on-shutdown configuration option.

Regardless of the HTTP method used, the endpoint always returns a response body with the ingester’s current unregister state:

Copy

{"unregister": true}

TSDB Metrics

Copy

GET /ingester/tsdb_metrics

This endpoint returns low-level metrics exposed by per-tenant TSDB. This can be useful for troubleshooting. Difference from /metrics is that metrics returned in /metrics are aggregated across all open TSDBs, while this endpoint returns TSDB metrics only for specific tenant.

Requires authentication, authenticated tenant is one whose TSDB metrics are returned.

Ingesters ring status

Copy

GET /ingester/ring

This endpoint displays a web page with the ingesters hash ring status, including the state, health, and last heartbeat time of each ingester.

Ingester tenants

Copy

GET /ingester/tenants

Displays a web page with the list of tenants with open TSDB on given ingester.

Ingester tenant TSDB

Copy

GET /ingester/tsdb/{tenant}

Displays a web page with details about tenant’s open TSDB on given ingester.

Querier / Query-frontend

The following endpoints are exposed both by the querier and query-frontend.

Instant query

Copy

GET,POST <prometheus-http-prefix>/api/v1/query

This endpoint is compatible with the Prometheus instant query endpoint.

For more information about Prometheus instant queries, refer to Prometheus instant query.

Requires authentication.

Range query

Copy

GET,POST <prometheus-http-prefix>/api/v1/query_range

This endpoint is compatible with the Prometheus range query endpoint. When a client sends a request through the query-frontend, the query-frontend uses caching and execution parallelization to accelerate the query.

For more information about Prometheus range queries, refer to Prometheus range query.

Requires authentication.

Exemplar query

Copy

GET,POST <prometheus-http-prefix>/api/v1/query_exemplars

This endpoint is compatible with the Prometheus exemplar query endpoint.

For more information about Prometheus exemplar queries, refer to Prometheus exemplar query.

Requires authentication.

Get series by label matchers

Copy

GET,POST <prometheus-http-prefix>/api/v1/series

For more information, refer to Prometheus series endpoint.

Requires authentication.

Get active series by selector

Copy

GET,POST <prometheus-http-prefix>/api/v1/cardinality/active_series

Returns the label sets of all active series matching a PromQL selector.

This endpoint is similar to the series endpoint but operates on the set of series considered active at the time of query processing. A series is considered active if any data has been written for it within the period specified by -ingester.active-series-metrics-idle-timeout.

This endpoint is disabled by default; you can enable it via the -querier.cardinality-analysis-enabled CLI flag (or its respective YAML configuration option).

Requires authentication.

Query parameters

• selector - mandatory - PromQL selector used to filter the result set.

Headers

• Sharding-Control - optional - Integer value specifying how many shards to use for request execution.
• X-Mimir-Response-Query-Stats - optional - Boolean value specifying if the response must have query stats in the Server-Timing header.

Response headers

• Server-Timing - optional - Contains query execution stats if the X-Mimir-Response-Query-Stats header is set to true. The content of the header is a comma-separated list of key-value pairs, where the key is the name of the metric and the value is the metric value. The contents of this header might change without notice. For example: Server-Timing: encode_time_seconds;dur=0.041, estimated_series_count;val=31950, fetched_chunk_bytes;val=23791904, fetched_chunks_count;val=329964, fetched_index_bytes;val=28647863, fetched_series_count;val=29906, query_wall_time_seconds;dur=6259.888, queue_time_seconds;dur=0.020, response_size_bytes;val=118, response_time;dur=6267.381, results_cache_hit_bytes;val=0, results_cache_miss_bytes;val=0, sharded_queries;val=0, split_queries;val=0
  • Metrics:
    • encode_time_seconds: The time, in seconds, spent at the frontend encoding the query’s final results. This value does not include time spent serializing results at the querier.
    • estimated_series_count: The estimated number of series to be fetched for the query.
    • fetched_chunk_bytes: The number of bytes of the chunks fetched for the query, after any deduplication.
    • fetched_chunks_count: The number of chunks fetched for the query, after any deduplication.
    • fetched_index_bytes: The number of index bytes fetched on the store-gateway for the query.
    • fetched_series_count: The number of series fetched for the query.
    • query_wall_time_seconds: The sum of all wall time, in seconds, spent in the querier to run the query.
    • queue_time_seconds: The sum of durations, in seconds, that the query spent in the queue, before being handled by querier.
    • response_size_bytes: The number of bytes of the response.
    • response_time: The time, in milliseconds, spent processing the query.
    • results_cache_hit_bytes: The number of results cache hits.
    • results_cache_miss_bytes: The number of results cache misses.
    • sharded_queries: The number of sharded queries run. Set to 0 if sharding is disabled, or if the query can’t be sharded.
    • split_queries: The number of split partial queries run. Set to 0 if splitting is disabled, or if the query can’t be split.

Response format

The response format is a subset of the series endpoint format including only the data field. The following shows an example request/response pair for this endpoint. Each item in the data array corresponds to a matched series.

shell Copy

$ curl 'http://localhost:9090/api/v1/cardinality/active_series' \
    --header 'Sharding-Control: 4' \ # optional
    --data-urlencode 'selector=up'

shell Copy

{
   "data" : [
      {
         "__name__" : "up",
         "job" : "prometheus",
         "instance" : "localhost:9090"
      },
      {
         "__name__" : "up",
         "job" : "node",
         "instance" : "localhost:9091"
      },
      {
         "__name__" : "process_start_time_seconds",
         "job" : "prometheus",
         "instance" : "localhost:9090"
      }
   ]
}

Caching

Responses for the active series endpoint are never cached.

Get label names

Copy

GET,POST <prometheus-http-prefix>/api/v1/labels

For more information, refer to Prometheus get label names.

Requires authentication.

Caching

The query-frontend can return a stale response fetched from the query results cache if -query-frontend.cache-results is enabled and -query-frontend.results-cache-ttl-for-labels-query set to a value greater than 0.

Get label values

Copy

GET <prometheus-http-prefix>/api/v1/label/{name}/values

For more information, refer to Prometheus get label values.

Requires authentication.

Caching

The query-frontend can return a stale response fetched from the query results cache if -query-frontend.cache-results is enabled and -query-frontend.results-cache-ttl-for-labels-query set to a value greater than 0.

Get metric metadata

Copy

GET <prometheus-http-prefix>/api/v1/metadata

Prometheus-compatible metric metadata endpoint.

For more information, refer to Prometheus metric metadata.

Requires authentication.

Remote read

Copy

POST <prometheus-http-prefix>/api/v1/read

Prometheus-compatible remote read endpoint.

For more information, refer to Prometheus Remote storage integrations.

Requires authentication.

Label names cardinality

Copy

GET,POST <prometheus-http-prefix>/api/v1/cardinality/label_names

Returns label names cardinality across all ingesters, for the authenticated tenant, in JSON format. It counts distinct label values per label name.

The items in the field cardinality are sorted by label_values_count in DESC order and by label_name in ASC order. The count of items is limited by limit request param.

This endpoint is disabled by default and can be enabled via the -querier.cardinality-analysis-enabled CLI flag (or its respective YAML config option).

Requires authentication.

Count series by inmemory or active

Two methods of counting are available: inmemory and active. To choose one, use the count_method parameter.

The inmemory method counts the labels in currently opened TSDBs in Mimir’s ingesters. Two subsequent calls might return completely different results if an ingester cut a block between calls. This method of counting is most useful for understanding ingester memory usage.

The active method also counts labels in currently opened TSDBs in Mimir’s ingesters, but filters out values that have not received a sample within a configurable duration of time. To configure this duration, use the -ingester.active-series-metrics-idle-timeout parameter. This method of counting is most useful for understanding what label values are represented in the samples ingested by Mimir in the last -ingester.active-series-metrics-idle-timeout. Two subsequent calls will likely return similar results, because this window of time is not related to the block cutting on ingesters. Values will change only as a result of changes in the data ingested by Mimir.

Caching

The query-frontend can return a stale response fetched from the query results cache if -query-frontend.cache-results is enabled and -query-frontend.results-cache-ttl-for-cardinality-query set to a value greater than 0.

Request params

• selector - optional - specifies PromQL selector that will be used to filter series that must be analyzed.
• count_method - optional - specifies which series counting method will be used. (default=“inmemory”, available options=[“inmemory”, “active”])
• limit - optional - specifies max count of items in field cardinality in response (default=20, min=0, max=500)

Response schema

JSON Copy

{
  "label_values_count_total": <number>,
  "label_names_count": <number>,
  "cardinality": [
    {
      "label_name": <string>,
      "label_values_count": <number>
    }
  ]
}

Label values cardinality

Copy

GET,POST <prometheus-http-prefix>/api/v1/cardinality/label_values

Returns label values cardinality associated to request param label_names[] across all ingesters, for the authenticated tenant, in JSON format. It returns the series count per label value associated to request param label_names[].

The items in the field labels are sorted by series_count in descending order and by label_name in ascending order. The items in the field cardinality are sorted by series_count in DESC order and by label_value in ASC order. The count of cardinality items is limited by request parameter limit.

This endpoint is disabled by default; you can enable it via the -querier.cardinality-analysis-enabled CLI flag (or its respective YAML configuration option).

Requires authentication.

Count series by inmemory or active

Two methods of counting are available: inmemory and active. To choose one, use the count_method parameter.

The inmemory method counts the number of series in currently opened TSDBs in Mimir’s ingesters. Two subsequent calls might return completely different results if an ingester cut a block between calls. This method of counting is most useful for understanding ingester memory usage.

The active method also counts series in currently opened TSDBs in Mimir’s ingesters, but filters out series that have not received a sample within a configurable duration of time. To configure this duration, use the -ingester.active-series-metrics-idle-timeout parameter. This method of counting is most useful for understanding what label values are represented in the samples ingested by Mimir in the last -ingester.active-series-metrics-idle-timeout. Two subsequent calls will likely return similar results, because this window of time is not related to the block cutting on ingesters. Values will change only as a result of changes in the data ingested by Mimir.

Caching

The query-frontend can return a stale response fetched from the query results cache if -query-frontend.cache-results is enabled and -query-frontend.results-cache-ttl-for-cardinality-query set to a value greater than 0.

Request params

• label_names[] - required - specifies labels for which cardinality must be provided.
• selector - optional - specifies PromQL selector that will be used to filter series that must be analyzed.
• count_method - optional - specifies which series counting method will be used. (default=“inmemory”, available options=[“inmemory”, “active”])
• limit - optional - specifies max count of items in field cardinality in response (default=20, min=0, max=500).

Response schema

JSON Copy

{
  "series_count_total": <number>,
  "labels": [
    {
      "label_name": <string>,
      "label_values_count": <number>,
      "series_count": <number>,
      "cardinality": [
        {
          "label_value": <string>,
          "series_count": <number>
        }
      ]
    }
  ]
}

• series_count_total - total number of series across opened TSDBs in all ingesters
• labels[].label_name - label name requested via the request param label_names[]
• labels[].label_values_count - total number of label values for the label name (note that dependent on the limit request param it is possible that not all label values are present in cardinality)
• labels[].series_count - total number of series having labels[].label_name
• labels[].cardinality[].label_value - label value associated to labels[].label_name
• labels[].cardinality[].series_count - total number of series having label_value for label_name

Querier

Get tenant ingestion stats

Copy

GET /api/v1/user_stats

Returns realtime ingestion rate, for the authenticated tenant, in JSON format.

Requires authentication.

Query-scheduler

Query-scheduler ring status

Copy

GET /query-scheduler/ring

Displays a web page with the query-scheduler hash ring status, including the state, healthy and last heartbeat time of each query-scheduler. The query-scheduler ring is available only when -query-scheduler.service-discovery-mode is set to ring.

Ruler

The ruler API endpoints require to configure a backend object storage to store the recording rules and alerts. The ruler API uses the concept of a “namespace” when creating rule groups. This is a stand in for the name of the rule file in Prometheus and rule groups must be named uniquely within a namespace.

Ruler ring status

Copy

GET /ruler/ring

Displays a web page with the ruler hash ring status, including the state, healthy and last heartbeat time of each ruler.

Ruler rules

Copy

GET /ruler/rule_groups

List all tenant rules. This endpoint is not part of ruler-API and is always available regardless of whether ruler-API is enabled or not. It should not be exposed to end users. This endpoint returns a YAML dictionary with all the rule groups for each tenant and 200 status code on success.

List Prometheus rules

Copy

GET <prometheus-http-prefix>/api/v1/rules?type={alert|record}&file={}&rule_group={}&rule_name={}&exclude_alerts={true|false}

Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded.

The type parameter is optional. If set, only the specified type of rule is returned.

The file, rule_group and rule_name parameters are optional, and can accept multiple values. If set, the response content is filtered accordingly. The parameters can also be provided as file[], rule_group[] and rule_name[] - if both are provided e.g file and file[] , file[] will take precdent.

The exclude_alerts parameter is optional. If set, it only returns rules and excludes active alerts.

The group_limit and group_next_token parameters are optional. If group_limit is set, it will limit the number of rule groups returned in a single response. If the total number of rule groups exceeds this value, the response will contain a groupNextToken. This can be passed into subsequent requests via group_next_token to paginate over the remaining groups. The final response will not contain a token. For more information, refer to Prometheus rules.

If both alerting and recording rule evaluation are disabled for the tenant, the endpoint returns an error with HTTP status code 422. If either alerting or recording rule evaluation is disabled for the tenant, the successful response includes warnings, that indicate that.

Requires authentication.

List Prometheus alerts

Copy

GET <prometheus-http-prefix>/api/v1/alerts

Prometheus-compatible rules endpoint to list all active alerts.

For more information, refer to Prometheus alerts documentation.

Requires authentication.

List rule groups

Copy

GET <prometheus-http-prefix>/config/v1/rules

List all rules configured for the authenticated tenant. This endpoint returns a YAML dictionary with all the rule groups for each namespace and 200 status code on success.

This endpoint can be disabled via the -ruler.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Example response

YAML Copy

---
<namespace1>:
- name: <string>
  interval: <duration;optional>
  source_tenants:
    - <string>
  rules:
  - record: <string>
      expr: <string>
  - alert: <string>
      expr: <string>
      for: <duration>
      annotations:
        <annotation_name>: <string>
      labels:
        <label_name>: <string>
- name: <string>
  interval: <duration;optional>
  source_tenants:
    - <string>
  rules:
  - record: <string>
      expr: <string>
  - alert: <string>
      expr: <string>
      for: <duration>
      annotations:
        <annotation_name>: <string>
      labels:
        <label_name>: <string>
<namespace2>:
- name: <string>
  interval: <duration;optional>
  source_tenants:
    - <string>
  rules:
  - record: <string>
      expr: <string>
  - alert: <string>
      expr: <string>
      for: <duration>
      annotations:
        <annotation_name>: <string>
      labels:
        <label_name>: <string>

Get rule groups by namespace

Copy

GET <prometheus-http-prefix>/config/v1/rules/{namespace}

Returns the rule groups defined for a given namespace. Escape the {namespace} path segment using percent-encoding, as defined by RFC 3986. For example, escape / to %2F.

This endpoint can be disabled via the -ruler.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Example response

YAML Copy

name: <string>
interval: <duration;optional>
source_tenants:
  - <string>
rules:
  - record: <string>
    expr: <string>
  - alert: <string>
    expr: <string>
    for: <duration>
    annotations:
      <annotation_name>: <string>
    labels:
      <label_name>: <string>

Get rule group

Copy

GET <prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}

Returns the rule group matching the request namespace and group name. Escape the {namespace} and {groupName} path segments using percent-encoding, as defined by RFC 3986. For example, escape / to %2F.

This endpoint can be disabled via the -ruler.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Set rule group

Copy

POST /<prometheus-http-prefix>/config/v1/rules/{namespace}

Creates or updates a rule group. This endpoint expects a request with Content-Type: application/yaml header and the rules group YAML definition in the request body, and returns 202 on success. The request body must contain the definition of one and only one rule group. Escape the {namespace} path segment using percent-encoding, as defined by RFC 3986. For example, escape / to %2F.

This endpoint can be disabled via the -ruler.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Example request body

YAML Copy

name: MyGroupName
rules:
  - alert: MyAlertName
    expr: up == 0
    labels:
      severity: warning

Delete rule group

Copy

DELETE /<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}

Deletes a rule group by namespace and group name. This endpoints returns 202 on success. Escape the {namespace} and {groupName} path segments using percent-encoding, as defined by RFC 3986. For example, escape / to %2F.

This endpoint can be disabled via the -ruler.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Delete namespace

Copy

DELETE /<prometheus-http-prefix>/config/v1/rules/{namespace}

Deletes all the rule groups in a namespace (including the namespace itself). This endpoint returns 202 on success. Escape the {namespace} path segment using percent-encoding, as defined by RFC 3986. For example, escape / to %2F.

This endpoint can be disabled via the -ruler.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Delete tenant configuration

Copy

POST /ruler/delete_tenant_config

This deletes all rule groups for a tenant, and returns 200 on success. Calling this endpoint when no rule groups exist for a tenant returns 200. Authentication is only to identify the tenant.

This is intended as internal API, and not to be exposed to users. This endpoint is enabled regardless of whether -ruler.enable-api is enabled or not.

Requires authentication.

Alertmanager

Alertmanager status

Copy

GET /multitenant_alertmanager/status

Displays a web page with the current status of the Alertmanager, including the Alertmanager cluster members.

Alertmanager configs

Copy

GET /multitenant_alertmanager/configs

List all Alertmanager configurations. This endpoint is not part of Alertmanager API and is always available regardless of whether Alertmanager API is enabled or not. It should not be exposed to end users. This endpoint returns a YAML dictionary with all the Alertmanager configurations and 200 status code on success.

Alertmanager ring status

Copy

GET /multitenant_alertmanager/ring

Displays a web page with the Alertmanager hash ring status, including the state, healthy and last heartbeat time of each Alertmanager instance.

Alertmanager UI

Copy

GET /<alertmanager-http-prefix>

Displays the Alertmanager UI.

Requires authentication.

Alertmanager Delete Tenant Configuration

Copy

POST /multitenant_alertmanager/delete_tenant_config

This endpoint deletes configuration for a tenant identified by X-Scope-OrgID header. It is internal, available even if Alertmanager API is disabled. The endpoint returns a status code of 200 if the user’s configuration has been deleted, or it didn’t exist in the first place.

Requires authentication.

Get Alertmanager configuration

Copy

GET /api/v1/alerts

Get the current Alertmanager configuration for the authenticated tenant, reading it from the configured object storage.

This endpoint doesn’t accept any URL query parameter and returns 200 on success.

This endpoint can be enabled and disabled via the -alertmanager.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Set Alertmanager configuration

Copy

POST /api/v1/alerts

Stores or updates the Alertmanager configuration for the authenticated tenant. The Alertmanager configuration is stored in the configured backend object storage.

This endpoint expects the Alertmanager YAML configuration in the request body and returns 201 on success.

The names of the templates in template_files must be valid file names and not contain any path separators. For example, both /templates/my-template.tpl and ./my-template.tpl are invalid, whereas my-template.tpl is valid.

This endpoint can be enabled and disabled via the -alertmanager.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Example request body

YAML Copy

template_files:
  default_template: |
    {{ define "__alertmanager" }}AlertManager{{ end }}
    {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }}
alertmanager_config: |
  global:
    smtp_smarthost: 'localhost:25'
    smtp_from: 'youraddress@example.org'
  templates:
    - 'default_template'
  route:
    receiver: example-email
  receivers:
    - name: example-email
      email_configs:
      - to: 'youraddress@example.org'

Delete Alertmanager configuration

Copy

DELETE /api/v1/alerts

Deletes the Alertmanager configuration for the authenticated tenant.

This endpoint doesn’t accept any URL query parameter and returns 200 on success.

This endpoint can be enabled and disabled via the -alertmanager.enable-api CLI flag (or its respective YAML config option).

Requires authentication.

Store-gateway

Store-gateway ring status

Copy

GET /store-gateway/ring

Displays a web page with the store-gateway hash ring status, including the state, healthy and last heartbeat time of each store-gateway.

Store-gateway tenants

Copy

GET /store-gateway/tenants

Displays a web page with the list of tenants with blocks in the storage configured for store-gateway.

Store-gateway tenant blocks

Copy

GET /store-gateway/tenant/{tenant}/blocks

Displays a web page listing the blocks for a given tenant.

Prepare for Shutdown

Copy

GET,POST,DELETE /store-gateway/prepare-shutdown

This endpoint changes in-memory store-gateway configuration to prepare for permanently stopping a store-gateway instance but does not actually stop any part of the latter.

After a POST to the prepare-shutdown endpoint returns, when the store-gateway process is stopped with SIGINT / SIGTERM, the store-gateway will be unregistered from the ring.

A GET to the prepare-shutdown endpoint returns the status of this configuration, either set or unset.

A DELETE to the prepare-shutdown endpoint reverts the configuration of the store-gateway to its previous state (with respect to unregistering).

This API endpoint is usually used by Kubernetes-specific scale down automations such as the rollout-operator.

Compactor

Compactor ring status

Copy

GET /compactor/ring

Displays a web page with the compactor hash ring status, including the state, healthy and last heartbeat time of each compactor.

Start block upload

Copy

POST /api/v1/upload/block/{block}/start

Starts the uploading of a TSDB block with a given ID to object storage. The client should send the block’s meta.json file as the request body. If the complete block already exists in object storage, a 409 (Conflict) status code gets returned. If the provided meta.json file is invalid, a 400 (Bad Request) status code gets returned. If the block’s max time is before the tenant’s retention period, a 422 (Unprocessable Entity) status code gets returned.

The provided meta.json file must have a thanos.files section with the list of the block’s files, otherwise the request will be rejected.

If the API request succeeds, a sanitized version of the block’s meta.json file gets uploaded to object storage as uploading-meta.json, and a 200 status code gets returned. Then you can start uploading files, and once done, you can request completion of the block upload.

Requires authentication.

Upload block file

Copy

POST /api/v1/upload/block/{block}/files?path={path}

Uploads a file with a given path, for a block with a given ID. The file path has to be one of the following, otherwise a 400 (Bad Request) status code gets returned:

• index
• chunks/<6-digit number>

The client must send the content of the file as the body of the request; if the body is empty, a 400 (Bad Request) status code gets returned. If the complete block already exists in object storage, a 409 (Conflict) status code gets returned. If an in-flight meta file (uploading-meta.json) doesn’t exist in object storage for the block in question, a 404 (Not Found) status code gets returned.

If the API request succeeds, the file gets uploaded with the given path to the block’s directory in object storage, and a 200 status code gets returned.

Requires authentication.

Complete block upload

Copy

POST /api/v1/upload/block/{block}/finish

Initiates the completion of a TSDB block with a given ID to object storage. If the complete block already exists in object storage, a 409 (Conflict) status code gets returned. If an in-flight meta file (uploading-meta.json) doesn’t exist in object storage for the block in question, a 404 (Not Found) status code gets returned. If the compactor has reached its limit for the maximum number of concurrent block upload validations, which is configured with -compactor.max-block-upload-validation-concurrency, a 429 (Too Many Requests) will be returned.

If the API request succeeds, compactor will start the block validation in the background. If the background validation passes block upload is finished by renaming in-flight meta file to meta.json in the block’s directory.

This API endpoint returns 200 (OK) at the beginning of the validation. To further check state of the block upload, use Check block upload API endpoint.

Requires authentication.

This API endpoint is experimental and subject to change.

Check block upload

Copy

GET /api/v1/upload/block/{block}/check

Returns state of the block upload. State is returned as JSON object with field result, with following possible values:

• complete – block validation is complete, and block upload is now finished.
• uploading – block is still being uploaded, and Complete block upload has not yet been called on the block.
• validating – block is being validated. Validation was started by call to Complete block upload API.
• failed – block validation has failed. Error message is available from error field of the returned JSON object.

Example response

JSON Copy

{ "result": "uploading" }

Example response

JSON Copy

{ "result": "failed", "error": "missing index file" }

Requires authentication.

This API endpoint is experimental and subject to change.

Tenant Delete Request

Copy

POST /compactor/delete_tenant

Request deletion of ALL tenant data for the tenant specified in the X-Scope-OrgID header. If authentication is disabled, then the default anonymous tenant is deleted (configurable by -auth.no-auth-tenant).

Requires authentication.

Tenant Delete Status

Copy

GET /compactor/delete_tenant_status

Returns status of tenant deletion.

Response schema

JSON Copy

{
  "tenant_id": "<id>",
  "blocks_deleted": true
}

The blocks_deleted field will be set to true if all the tenant’s blocks have been deleted.

Requires authentication.

Compactor tenants

Copy

GET /compactor/tenants

Displays a web page with the list of tenants that have blocks in the storage configured for the compactor.

Compactor tenant planned jobs

Copy

GET /compactor/tenant/{tenant}/planned_jobs

Displays a web page listing planned compaction jobs computed from the bucket index for the given tenant.

Overrides-exporter

Overrides-exporter ring status

Copy

GET /overrides-exporter/ring

Displays a web page with the overrides-exporter hash ring status, including the state, healthy and last heartbeat time of each overrides-exporter. The overrides-exporter ring is available only when -overrides-exporter.ring.enabled is set to true.