This is documentation for the next version of Grafana Mimir documentation. For the latest stable release, go to the latest version.

Release notes

V3.1 release notes

Open source

Grafana Mimir version 3.1 release notes

Grafana Labs is excited to announce version 3.1 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Grafana Mimir version 3.1 includes the following key features and enhancements.

More Kafka options for Ingest storage

Ingest storage now supports additional ways to authenticate with Kafka through the new -ingest-storage.kafka.sasl-mechanism flag, including SCRAM, OAUTHBEARER, and AWS MSK IAM authentication. In addition, new -ingest-storage.kafka.tls* flags allow connecting to Kafka clusters over TLS, including mTLS.

You can also configure multiple Kafka seed brokers via comma-separated values in -ingest-storage.kafka.address and enable rack-aware consumption with -ingest-storage.kafka.client-rack.

Separate ingestion limits by tenant metadata

Distributors can now track limits separately based on tenant metadata. This allows operators to track limits separately for subsets of write requests belonging to the same tenant, for example to prioritize some sources of metrics over others.

Clients may pass tenant metadata in the X-Scope-OrgID header using the format tenantID:key1=value1:key2=value2, and operators may define per-metadata overrides in the runtime configuration.

Mimir Query Engine (MQE) improvements

MQE continues to receive significant optimizations in this release:

Support for experimental PromQL extended range selector modifiers smoothed and anchored, enabled with -query-frontend.enabled-promql-extended-range-selectors=true.
Optimization passes for common subexpression elimination, subset selector elimination, projection pushdown, and multi-aggregation without buffering.
Improved per-query memory consumption limit enforcement in a variety of scenarios.
Experimental support for splitting and caching intermediate results for functions over range vectors in instant queries.
Support for the experimental info() PromQL function.

Zone-aware memberlist routing

A new experimental zone-aware routing feature for memberlist reduces cross-AZ data transfer by routing gossip messages within the local availability zone when possible. Configure it with -memberlist.zone-aware-routing.* flags.

Additional improvements

Grafana Mimir 3.1 also includes:

Store-gateways now verify CRC32 checksums for 1 out of every 128 chunks read from object storage and the chunks cache to detect corruption.
GCS uploads are now optionally retryable, with configurable max retries per storage backend.
Disk interaction has been removed when loading ruler rules. Rule evaluation failures now include a reason label (operator or user) in prometheus_rule_evaluation_failures_total for better error classification.
Query blocking via blocked_queries is now stable and no longer experimental, with support for blocking queries exceeding a time range duration (time_range_longer_than) or with steps smaller than a threshold (step_size_shorter_than).
The per-tenant postings-for-matchers cache is now stable.
Out-of-order ingestion support is now stable, configured via -ingester.out-of-order-time-window.
The -alertmanager.utf8-strict-mode-enabled flag is now stable.
The query-scheduler now drains the queue before exiting during shutdown.
Distributors support zone-aware rate limiting via -distributor.ring.instance-availability-zone, dividing the global ingestion rate by zones instead of total distributors.
Default ingest storage configuration now enables concurrency settings for improved throughput.
Optional per-tenant max limits for label name and label value requests via max_label_names_limit and max_label_values_limit.
Runtime configuration can now be loaded from HTTP URLs in addition to local files via -runtime-config.file. This may reduce configuration propagation times.
Blocked queries configuration is now validated at load time.

Important changes

Grafana Mimir 3.1 introduces several updates that change default behavior and configuration. Review these changes before upgrading:

Experimental support for disabling ring heartbeats and heartbeat timeouts has been removed.
The -target=flusher mode has been removed; use the /ingester/flush HTTP endpoint instead.
Uploaded TSDB blocks must now use v2 of the index file format. Store-gateways no longer generate index-headers from v1 index format blocks.
Per-step stats are no longer supported when MQE is enabled. The -query-frontend.cache-samples-processed-stats flag is deprecated and has no effect.
The -querier.response-streaming-enabled flag has been removed; active series responses are now always streamed.
cortex_ingest_storage_writer_buffered_produce_bytes has been renamed to cortex_ingest_storage_writer_buffered_produce_bytes_distribution.
Metric cortex_ingester_owned_target_info_series has been removed.
The cost_attribution_labels configuration option has been removed; use cost_attribution_labels_structured instead.
-querier.prefer-availability-zone has been renamed to -querier.prefer-availability-zones and now accepts a comma-separated list.
The per-query memory consumption limit now considers more sources of memory consumption. As a result, queries that previously succeeded may now fail due to exceeding the memory consumption limit.
The following flags have been removed:
- -distributor.metric-relabeling-enabled
- -compactor.no-blocks-file-cleanup-enabled
- -compactor.in-memory-tenant-meta-cache-size
- -blocks-storage.bucket-store.index-header.eager-loading-startup-enabled
- *.memcached.dns-ignore-startup-failures

Experimental features

Grafana Mimir 3.1 includes some features that are experimental. Use these features with caution and report any issues that you encounter:

New usage-tracker component to enforce series limits before data is ingested.
Zone-aware memberlist routing to reduce cross-AZ data transfer.
Query planning in query-frontends with distributed execution across queriers.
Support in MQE for experimental PromQL extended range selector modifiers (smoothed, anchored).
Support in MQE for the experimental info() PromQL function.
MQE optimization passes: multi-aggregation, subset selector elimination, common subexpression elimination for range vector expressions.
Per-zone store-gateway shard size (-store-gateway.tenant-shard-size-per-zone).
Running ingesters with no tokens in the ring when ingest storage is enabled (-ingester.ring.num-tokens=0).
Per-sample HA deduplication (-distributor.ha-tracker.per-sample-dedupe).
Per-tenant early head compaction for ingesters based on owned series count.
Store-gateway excluded zones (-store-gateway.sharding-ring.excluded-zones).
Controlling OTLP metric name suffix addition and translation strategy via request headers, gated by -api.otlp-translation-headers-enabled.
Memberlist propagation delay tracker (-memberlist.propagation-delay-tracker.enabled).
Reporting the number of samples read per query in MQE.