This is documentation for the next version of Mimir. For the latest stable release, go to the latest version.

V2.16 release notes

Open source

Grafana Mimir version 2.16 release notes

Grafana Labs is excited to announce version 2.16 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release. For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

In rulers, when rule concurrency is enabled for a rule group, its rules are now reordered and run in batches based on their dependencies. This increases the number of rules that can run concurrently. Note that the global and tenant-specific limits around the number of rule groups and rules per group still apply.

Using mimirtool to analyze Grafana dashboards now supports bar chart, pie chart, state timeline, status history, histogram, candlestick, canvas, flame graph, geomap, node graph, trend, and XY chart panels.

Important changes

In Grafana Mimir 2.16, the following behavior has changed:

Grafana Mimir only provides container images based on distroless images. Alpine Linux-based container images were deprecated in the 2.12 release and are no longer built.

How experimental PromQL functions are enabled has changed.

The experimental CLI flags -querier.promql-experimental-functions-enabled and -query-frontend.block-promql-experimental-functions and their respective YAML configurations have been removed from query-frontends and queriers.
Experimental PromQL functions are disabled by default. You can enable them using only the enabled_promql_experimental_functions per-tenant setting.

Support for native histograms and out-of-order native histograms is enabled by default in ingesters.

Distributors discard float and histogram samples with duplicated timestamps from each timeseries in a request before the request is forwarded to ingesters. Discarded samples are tracked by cortex_discarded_samples_total metrics with the reason sample_duplicate_timestamp.

Experimental features

Grafana Mimir 2.16 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

Distributors now include experimental support for the Influx line protocol.

Query-frontends now include experimental support to “spin off” subqueries as actual range queries, so that they benefit from query acceleration techniques such as sharding, splitting, and caching.

Bug fixes

Distributor: Use a boolean to track changes while merging the ReplicaDesc components, rather than comparing the objects directly. #10185
Querier: Fix timeout responding to the query-frontend when the response size is within a few hundred bytes of -querier.frontend-client.grpc-max-send-msg-size. #10154
Query-frontend and querier: Show warning and info annotations in some cases where they were missing (if a lazy querier was used). #10277
Query-frontend: Fix an issue where transient errors are inadvertently cached. #10537 #10631
Ruler: Fix indeterminate rules always, instead of never, running concurrently when -ruler.max-independent-rule-evaluation-concurrency is set. https://github.com/prometheus/prometheus/pull/15560 #10258
PromQL: Fix various UTF-8 bugs related to quoting. https://github.com/prometheus/prometheus/pull/15531 #10258
Ruler: Fix an issue when using the experimental -ruler.max-independent-rule-evaluation-concurrency feature, where if a rule group was eligible for concurrency, it would flap between running concurrently or not based on the time it took after running concurrently. #9726 #10189
Mimirtool: remote-read commands now return data. #10286
PromQL: Fix deriv, predict_linear and double_exponential_smoothing with histograms https://github.com/prometheus/prometheus/pull/15686 #10383
MQE: Fix deriv with histograms. #10383
PromQL: Fix <aggr_over_time> functions with histograms. https://github.com/prometheus/prometheus/pull/15711 #10400
MQE: Fix <aggr_over_time> functions with histograms. #10400
Distributor: Return HTTP status 415, Unsupported Media Type, instead of 200, Success, for Remote Write 2.0 until we support it. #10423 #10916
Query-frontend: Add -query-frontend.prom2-range-compat flag and corresponding YAML to rewrite queries with ranges that worked in Prometheus 2 but are invalid in Prometheus 3. #10445 #10461 #10502
Distributor: Fix edge case at the HA-tracker with memberlist as KVStore, where when a replica in the KVStore is marked as deleted but not yet removed, it fails to update the KVStore. #10443
Distributor: Fix panics in DurationWithJitter util functions when computed variance is zero. #10507
Ingester: Fixed a race condition in the PostingsForMatchers cache that may have infrequently returned expired cached postings. #10500
Distributor: Report partially converted OTLP requests with status 400, Bad Request. #10588
Ruler: Fix issue where rule evaluations could be missed while shutting down a ruler instance if that instance owns many rule groups. prometheus/prometheus#15804 #10762
Ingester: Add additional check on reactive limiter queue sizes. #10722
TSDB: Fix unknown series errors and possible lost data during WAL replay when series are removed from the head due to inactivity and reappear before the next WAL checkpoint. https://github.com/prometheus/prometheus/pull/16060 #10824
Querier: Fix issue where label_join could incorrectly return multiple series with the same labels rather than failing with vector cannot contain metrics with the same labelset. https://github.com/prometheus/prometheus/pull/15975 #10826
Querier: Fix issue where counter resets on native histograms could be incorrectly under or over-counted when using subqueries. https://github.com/prometheus/prometheus/pull/15987 #10871
Ingester: Fix goroutines and memory leak when experimental ingest storage is enabled and a server-side error occurs during metrics ingestion. #10915
Mimirtool: Fix issue where MIMIR_HTTP_PREFIX environment variable was ignored and the value from MIMIR_MIMIR_HTTP_PREFIX was used instead. #10207