Grafana Mimir operator and user guideGrafana Mimir toolsGrafana mimir-continuous-test

Grafana mimir-continuous-test

As a developer, you can use the standalone mimir-continuous-test tool to continuously run smoke tests on live Grafana Mimir clusters. This tool identifies a class of bugs that could be difficult to spot during development.

Download mimir-continuous-test

  • Using Docker:
docker pull "grafana/mimir-continuous-test:latest"
  • Using a local binary:

Download the appropriate mimir-continuous-test binary for your operating system and architecture, and make it executable.

For Linux with the AMD64 architecture, execute the following command:

curl -Lo mimir-continuous-test
chmod +x mimir-continuous-test

Configure mimir-continuous-test

Mimir-continuous-test requires the endpoints of the backend Grafana Mimir clusters and the tenant ID for writing and querying testing metrics:

  • Set -tests.write-endpoint to the base endpoint on the write path. Remove any trailing slash from the URL. The tool appends the specific API path to the URL, for example /api/v1/push for the remote-write API.
  • Set to the base endpoint on the read path. Remove any trailing slash from the URL. The tool appends the specific API path to the URL, for example /api/v1/query_range for the range-query API.
  • Set -tests.tenant-id to the tenant ID to use to write and read metrics in tests.

Note: You can run mimir-continuous-test -help to list all available configuration options.

How it works

Mimir-continuous-test periodically runs a suite of tests, writes data to Mimir, queries that data back, and checks if the query results match what is expected. The tool exposes metrics that you can use to alert on test failures, and the tool logs the details about the failed tests.

Exported metrics

Mimir-continuous-test exposes the following Prometheus metrics at the /metrics endpoint listening on the port that you configured via the flag -server.metrics-port:

# HELP mimir_continuous_test_writes_total Total number of attempted write requests.
# TYPE mimir_continuous_test_writes_total counter

# HELP mimir_continuous_test_writes_failed_total Total number of failed write requests.
# TYPE mimir_continuous_test_writes_failed_total counter

# HELP mimir_continuous_test_queries_total Total number of attempted query requests.
# TYPE mimir_continuous_test_queries_total counter

# HELP mimir_continuous_test_queries_failed_total Total number of failed query requests.
# TYPE mimir_continuous_test_queries_failed_total counter

# HELP mimir_continuous_test_query_result_checks_total Total number of query results checked for correctness.
# TYPE mimir_continuous_test_query_result_checks_total counter

# HELP mimir_continuous_test_query_result_checks_failed_total Total number of query results failed when checking for correctness.
# TYPE mimir_continuous_test_query_result_checks_failed_total counter


Grafana Mimir alerts include checks on failures that mimir-continuous-test tracks. When running mimir-continuous-test, use the provided alerts.