Grafana Mimir release notes on Grafana Labs

Grafana Mimir version 2.0 release notes

Mon, 17 Mar 2025 08:45:08 +0100

Grafana Mimir version 2.0 release notes

Grafana Labs is excited to announce the first release of Grafana Mimir, the most scalable, most performant open source time series database in the world. In customer tests, we’ve shown that a single cluster can support more than one billion active time series.

Besides massive scale, Grafana Mimir offers a host of other benefits, including easy deployment, native multi-tenancy, high availability, durable long-term storage, and exceptional query performance on even the highest-cardinality queries.

For more information about Grafana Mimir features and capabilities, watch the Introducing Grafana Mimir video.

{ responded = true; response && response.status === 200 ? vimeo_is_up = true : vimeo_is_up = false; }) .catch(error => { responded = true; })" >

We’re launching Grafana Mimir with a 2.0 version number to signal our respect for Cortex, the project from which Grafana Mimir was forked. The choice of 2.0 also represents our conviction that Grafana Mimir is real-world-tested, production-ready software. It has served as the backbone of our Grafana Cloud Metrics and Grafana Enterprise Metrics products since their inception.

For more on the story of why we created Grafana Mimir, please see the following blog posts:

Grafana Mimir builds on Cortex 1.10.0, adding features we developed to run Grafana Enterprise Metrics and Grafana Cloud Metrics at massive scale. As a result, these listed features and enhancements, migration considerations, and bug fixes highlight changes between Cortex 1.10.0 and Grafana Mimir 2.0.

The complete list of changes is recorded in the Changelog.

Features and enhancements

These features and enhancements distinguish Grafana Mimir from Cortex 1.10.0:

Simplified deployment experience: These changes improve the experience of setting up and maintaining Grafana Mimir:
- We revamped Cortex’s single process mode, renaming it monolithic mode. Monolithic mode now runs the required compactor component, and has better configuration defaults, so that it works with fewer changes out-of-the-box.
- We switched to memberlist as the default store for Grafana Mimir’s hash rings. With this, users no longer have to run Consul or etcd as external dependencies. We made performance optimizations to memberlist to reduce its CPU utilization, which ensures that memberlist runs smoothly on Grafana Mimir clusters with lots of active series.
- We’ve included our own internal best practice dashboards, mixins, and alerts for monitoring Grafana Mimir. While installing monitoring best practices such as these has typically required use of Jsonnet, we’ve eliminated this requirement. We include dashboards as JSON and alerting and recording rules as YAML which can be directly imported into your Grafana and Prometheus deployments. The alerts are accompanied by runbooks distilled from our own internal operations.
Configuration parameter reduction and classification: We removed 36% of the configuration parameters in Grafana Mimir. All remaining configuration parameters have been classified as basic, advanced, or experimental. This is meant to make Grafana Mimir’s configuration more approachable for new users. In a default installation, you can focus exclusively on basic configuration. As you become more familiar with Grafana Mimir and want to push your clusters further, you can choose to tune advanced parameters or use experimental parameters. Refer to parameter categories to learn more.
More-scalable metrics compaction: Grafana Mimir’s new compactor uses a split-and-merge compaction algorithm. This new compactor parallelizes the compaction of overlapping blocks across multiple machines, and splits blocks in such a way as to overcome Prometheus’ 64 GB index size limit. This solves a problem that we’ve seen with tenants with greater than 30M time series in Cortex - namely, that the count of uncompacted blocks grew indefinitely because the compactor couldn’t work fast enough. With its new compactor, a single Grafana Mimir cluster can support over one billion active time series. Refer to compactor to learn more.
Query sharding for improved query speeds: Grafana Mimir introduces query sharding to accelerate the execution of high-cardinality or CPU-intensive queries. Query sharding distributes the execution of a single query across multiple machines, to significantly reduce query execution time. We have seen speedups of 10 to 30x in our Grafana Cloud Metrics clusters. Refer to query sharding to learn more.
Federated rule groups: Grafana Mimir makes it possible to write alerting and recording rules that use metrics data from multiple tenants. For example, a user can now create a recording rule that adds metricA from tenantA to metricB from tenantB and writes the result to tenantC. This feature is experimental. For more information on how to use it, refer to federated rule groups.
Understand your metrics cardinality: Grafana Mimir adds two API endpoints to help users identify high-cardinality metrics. The label_names endpoint takes a metric name and returns all label names applied to that metric, as well as the count of values for each label name. When run without a metric name, it returns the highest-cardinality label names. The label_values endpoint returns the highest-cardinality metrics, and can be used to get a count of how many series have a given label-value pair applied. The new custom tracker feature takes this one step further by allowing you to track the count of active series over time that match a specific label matcher.

These features are no longer considered experimental; they are stable features:

Cross tenant query federation is a feature that allows you to issue queries that aggregate series across multiple tenants, providing a way to get a global view of your data. Refer to tenant-federation.enabled in the configuration parameters block to learn more.
Zone-aware replication protects against data loss due to an outage in a specific zone by ensuring data is replicated across multiple zones. In addition to promoting this feature to stable, we’ve also released a Kubernetes operator that makes it easier for those running Grafana Mimir on Kubernetes to manage multi-zone rollouts. Refer to Configuring zone-aware replication to learn more.
Azure Blob storage and OpenStack Swift support: Two additional object storage options are available. Grafana Mimir now supports AWS S3, GCS, Azure Blob Storage, OpenStack Swift, as well as any S3-compatible object storage.
Horizontally scalable Alertmanager: The Alertmanager can distribute its workload among multiple Alertmanager replicas, increasing the maximum number of tenants a cluster can comfortably handle. The Alertmanager uses the tenant ID to determine which replica(s) own which tenant. The replication factor is configurable, so alerts are replicated across multiple replicas, reducing the likelihood of a dropped alert.
Query-scheduler component: The query-scheduler component solves the scalability limitations of the query-frontend component. We strongly encourage users operating larger Grafana Mimir clusters to run this optional component. Refer to the query-scheduler to learn more about this component.

Migration considerations

This information is relevant only to users migrating from Cortex to Grafana Mimir. Those who are not currently using Cortex today can follow our tutorial to Get Mimir up and running in 10 minutes.

Grafana Mimir is 100% compatible with Cortex, and the migration from Cortex to Mimir can be done without any data loss or downtime.

Grafana Mimir 2.0 is a major version upgrade, and therefore it introduces several breaking changes for those migrating from Cortex. To make this upgrade as easy as possible, we’ve added functionality to mimirtool to automate the required configuration changes. We’ve paired this tooling with detailed migration guides that walk through all required changes.

For a demonstration that illustrates how easy this migration can be, please see the following video:

{ responded = true; response && response.status === 200 ? vimeo_is_up = true : vimeo_is_up = false; }) .catch(error => { responded = true; })" >

Bug fixes

2.0.0 bug fixes

Version 2.0.0 fixes these bugs:

PR 424: Fixed a bug where cached results for step-aligned queries were used to respond to unaligned queries, causing the user to receive incorrect results. The new configuration parameter query-frontend.cache-unaligned-requests safely enables the caching of query results, even when the query step is not aligned.
PR 224: Fixed a race condition in the Alertmanager, which sometimes resulted in a tenant’s configuration being replaced with the blank fallback configuration.
PR 551: Fixed a bug in memberlist that caused large messages to be corrupted. This bug fix, in addition to the memberlist performance optimizations, enabled us to switch to memberlist as the default store for Grafana Mimir’s hash rings.

Grafana Mimir version 2.1 release notes

Mon, 17 Mar 2025 08:45:08 +0100

Grafana Mimir version 2.1 release notes

Grafana Labs is excited to announce version 2.1 of Grafana Mimir, the most scalable, most performant open source time series database in the world.

Below we highlight the top features, enhancements and bugfixes in this release, as well as relevant callouts for those upgrading from Grafana Mimir 2.0. The complete list of changes is recorded in the Changelog.

Features and enhancements

Mimir on ARM: We now publish Docker images for both amd64 and arm64, making it easier for those on arm-based machines to develop and run Mimir. Multiplaform images are available from the the Mimir docker registry. Note that our existing integration test suite only uses the amd64 images, which means we cannot make any functional or performance guarantees about the arm64 images.
Remote ruler mode for improved rule evaluation performance: We’ve added a remote mode for the Grafana Mimir ruler, in which the ruler delegates rule evaluation to the query-frontend rather than evaluating rules directly within the ruler process itself. This allows recording and alerting rules to benefit from the query parallelization techniques implemented in the query-frontend (like query sharding). Remote mode is considered experimental and is off by default. To enable, see remote ruler.
Per-tenant custom trackers for monitoring cardinality: In Grafana Mimir 2.0, we introduced a custom tracker feature that allows you to track the count of active series over time that match a specific label matcher. In Grafana Mimir 2.1, we’ve made it possible to configure custom trackers via the runtime configuration file. This means you can now define different trackers for each tenant in your cluster and modify those trackers without an ingester restart.
Reduce cardinality of Grafana Mimir’s /metrics endpoint: While Grafana Mimir does a good job of exposing a relatively small number of series about its own state, this number can tick up when running Grafana Mimir clusters with high tenant counts or high active series counts. To reduce this number (and the accompanying cost of scraping and storing these time series), we made several optimizations which decreased series count on the /metrics endpoint by more than 10%.

Upgrade considerations

We’ve updated the default values for 2 parameters in Grafana Mimir to give users better out-of-the-box performance:

We’ve changed the default for -blocks-storage.tsdb.isolation-enabled from true to false. We’ve marked this flag as deprecated and will remove it completely in 2 releases. TSDB isolation is a feature inherited from Prometheus that didn’t provide any benefit given Grafana Mimir’s distributed architecture and in our 1 billion series load test we found it actually hurt performance. Disabling it reduced our ingester 99th percentile latency by 90%.
The store-gateway attributes cache is now enabled by default (achieved by updating the default for -blocks-storage.bucket-store.chunks-cache.attributes-in-memory-max-items from 0 to 50000). This in-memory cache makes it faster to look up object attributes for chunk data. We’ve been running this optional cache internally for a while and upon a recent configuration audit, realized it made sense to do the same for all users. The increase in store-gateway memory utilization from enabling this cache is negligible and easily justified given the performance gains.
As part of the enhancement to make active series custom trackers configurable on a per-tenant basis, we’ve moved where they get configured. Users of this feature should migrate from defining active_series_custom_trackers_config in the ingester section of Grafana Mimir’s YAML configuration to defining it in the limits section. Both options are supported for now to give users time to make the change, but configuring active_series_custom_trackers_config via the ingester section has been marked deprecated and will be removed in Mimir 2.3. Note: Configuring active_series_custom_trackers_config in both the limits and ingester sections will cause Grafana Mimir to fail on startup.

Bug fixes

2.1.0 bug fixes

PR 1704: Fixed a bug that previously caused Grafana Mimir to crash on startup when trying to run in monolithic mode with the results cache enabled due to duplicate metric names.
PR 1835: Fixed a bug that caused Grafana Mimir to crash when an invalid Alertmanager configuration was set even though the Alertmanager component was disabled. After this fix, the Alertmanager configuration is only validated if the Alertmanager component is loaded.
PR 1836: The ability to run Alertmanager with local storage broke in Grafana Mimir 2.0 when we removed the ability to run the Alertmanager without sharding. With this bugfix, we’ve made it possible to again run Alertmanager with local storage. However, for production use, we still recommend using external store since this is needed to persist Alertmanager state (e.g. silences) between replicas.
PR 1715: Restored Grafana Mimir’s ability to use CNAME DNS records to reach memcached servers. The bug was inherited from an upstream change to Thanos; we contributed a fix to Thanos and subsequently updated our Thanos version.

Grafana Mimir version 2.2 release notes

Mon, 17 Mar 2025 08:45:08 +0100

Grafana Mimir version 2.2 release notes

Grafana Labs is excited to announce version 2.2 of Grafana Mimir, the most scalable, most performant open source time series database in the world.

The highlights that follow include the top features, enhancements, and bugfixes in this release. If you are upgrading from Grafana Mimir 2.1, there is upgrade-related information as well. For the complete list of changes, see the Changelog.

Features and enhancements

Support for ingesting out-of-order samples: Grafana Mimir includes new, experimental support for ingesting out-of-order samples. This support is configurable, and it allows you to set how far out-of-order Mimir accepts samples on a per-tenant basis. This feature still needs additional testing; we do not recommend using it in a production environment. For more information, see Configuring out-of-order samples ingestion
Improved error messages: The error messages that Mimir reports are more human readable, and the messages include error codes that are easily searchable. For error descriptions, see the Grafana Mimir runbooks’ Errors catalog.
Configurable prefix for object storage: Mimir can now store block data, rules, and alerts in one bucket, with each under its own user-defined prefix, rather than requiring one bucket for each. You can configure the storage prefix by using -<storage>.storage-prefix option for corresponding storage: ruler-storage, alertmanager-storage or blocks-storage.
Store-gateway performance optimization The store-gateway can now pre-populate the file system cache when memory-mapping index-header files. This avoids the store-gateway from appearing to be stuck while loading index-headers. This feature is experimental and disabled by default; enable it using the flag -blocks-storage.bucket-store.index-header.map-populate-enabled.
Faster ingester startup: Ingesters now replay their WALs (write ahead logs) about 50% faster, and they also re-join the ring sooner under some conditions.
Helm Chart improvements: The Mimir Helm chart is the best way to install Mimir on Kubernetes. As part of the Mimir 2.2 release, we’re also releasing version 3.0 of the Helm chart. Notable enhancements follow. For the full list of changes, see the Helm chart changelog.
- The Helm chart now supports OpenShift.
- The Helm chart can now easily deploy Grafana Agent in order to scrape metrics and logs from all Mimir pods, and ship them to a remote store, which makes it easier to monitor the health of your Mimir installation. For more information, see Collecting metrics and logs from Grafana Mimir.
- The Helm chart now enables multi-tenancy by default. This makes it easy for you to add tenants as you grow your cluster. You can take advantage of Mimir’s per-tenant quality-of-service features, which improves stability and resilience at high scale. To learn more about how multi-tenancy in Mimir works, see Grafana Mimir authorization and authentication. This change is backwards-compatible. To read about how we implemented this, see #2117.
- We have significantly improved the configuration experience for the Helm chart, and here are a few of the most salient changes:
  - We’ve added an extraEnvFrom capability to all Mimir services to enable you to inject secrets via environment variables.
  - We’ve made it possible to globally set environment variables and inject secrets across all pods in the chart using global.extraEnv and global.extraEnvFrom. Note that the memcached and minio pods are not included.
  - We’ve switched the default storage of the Mimir configuration from a Secret to a ConfigMap, which makes it easier to quickly see the differences between your Mimir configurations between upgrades. We especially like the Helm diff plugin for this purpose.
  - We’ve added a structuredConfig option, which allows you to overwrite specific key-value pairs in the mimir.config template, which saves you from having to maintain the entire mimir.config in your own values.yaml file.
  - We’ve added the ability to create global pod annotations. This unlocks the ability to trigger a restart of all services in response to a single event, such as the update of the secret containing Mimir’s storage credentials.
- We’ve set the chart to disable -ingester.ring.unregister-on-shutdown and -distributor.extend-writes, for a smoother upgrade experience. Rolling restarts of ingesters are now less likely to cause spikes in resource usage.
- We’ve improved the documentation for the Helm chart by adding a Get started with Mimir using the Helm chart guide.
- We’ve added a smoke test for your Mimir cluster to help catch errors immediately after you install or upgrade Mimir via the Helm chart.

Upgrade considerations

All deprecated API endpoints that are under /api/v1/rules* and /prometheus/rules* have now been removed from the ruler component in favor of identical endpoints that use the prefix /prometheus/config/v1/rules*.

In Grafana Mimir 2.2, we have updated default values and some parameters to give you a better out-of-the-box experience:

Message size limits for gRPC messages that are exchanged between internal Mimir components have increased to 100 MiB from 4 MiB. This helps to avoid internal server errors when pushing or querying large data.
The -blocks-storage.bucket-store.ignore-blocks-within parameter changed from 0 to 10h. The default value of -querier.query-store-after changed from 0 to 12h. For most-recent data, both changes improve query performance by querying only the ingesters, rather than object storage.
The option -querier.shuffle-sharding-ingesters-lookback-period has been deprecated. If you previously changed this option from its default of 0s, set -querier.shuffle-sharding-ingesters-enabled to true and specify the lookback period by setting the -querier.query-ingesters-within option.
The -memberlist.abort-if-join-fails parameter now defaults to false. When Mimir is using memberlist as the backend store for its hash ring, and it fails to join the memberlist cluster, Mimir no longer aborts startup by default.

If you have used a previous version of the Mimir Helm chart, you must address some of the chart’s breaking changes before upgrading to helm chart version 3.0. For a detailed information about how to do this, see the Upgrade the Grafana Mimir Helm chart from version 2.1 to 3.0 guide.

Bug fixes

PR 1883: Fixed a bug that caused the query-frontend and querier to crash when they received a user query with a special regular expression label matcher.
PR 1933: Fixed a bug in the ingester ring page, which showed incorrect status of entries in the ring.
PR 2090: Ruler in remote rule evaluation mode now applies the timeout correctly. Previously the ruler could get stuck forever, which halted rule evaluation.
PR 2036: Fixed panic at startup when Mimir is running in monolithic mode and query sharding is enabled.

Grafana Mimir version 2.3 release notes

Mon, 17 Mar 2025 08:45:08 +0100

Grafana Mimir version 2.3 release notes

Grafana Labs is excited to announce version 2.3 of Grafana Mimir, the most scalable, most performant open source time series database in the world.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Note: If you are upgrading from Grafana Mimir 2.2, review the list of important changes that follow.

Features and enhancements

Ingest metrics in OpenTelemetry format: This release of Grafana Mimir introduces experimental support for ingesting metrics from the OpenTelemetry Collector’s otlphttp exporter. This adds a second ingestion option for users of the OTel Collector; Mimir was already compatible with the prometheusremotewrite exporter. For more information, please see Configure OTel Collector.
Tenant federation for metadata queries: Users with tenant federation enabled could already issue instant queries, range queries, and exemplar queries to multiple tenants at once and receive a single aggregated result. With Grafana Mimir 2.3, we’ve added tenant federation support to the /api/v1/metadata endpoint as well.
Simpler object storage configuration: Users can now configure block, alertmanager, and ruler storage all at once with the common YAML config option key (or -common.storage.* CLI flags). By centralizing your object storage configuration in one place, this enhancement makes configuration faster and less error prone. Users may still individually configure storage for each of these components if they desire. For more information, see the Common Configurations.
.deb and .rpm packages for Mimir: Starting with version 2.3, we’re publishing .deb and .rpm files for Grafana Mimir, which will make installing and running it on Debian or RedHat-based linux systems much easier. Thank you to community contributor wilfriedroset for your work to implement this!
Import historic data: Users can now backfill time series data from their existing Prometheus or Cortex installation into Mimir using mimirtool, making it possible to migrate to Grafana Mimir without losing your existing metrics data. This support is still considered experimental and does not yet work for data stored in Thanos. To learn more about this feature, please see mimirtool backfill and Configure TSDB block upload
Increased instant query performance: Grafana Mimir now supports splitting instant queries by time. This allows it to better parallelize execution of instant queries and therefore return results faster. At present, splitting is only supported for a subset of instant queries, which means not all instant queries will see a speedup. This feature is currently experimental and is disabled by default. It can be enabled with the split_instant_queries_by_interval YAML config option in the limits section (or the CLI flag -query-frontend.split-instant-queries-by-interval).

Helm chart improvements

The Mimir Helm chart is the best way to install Mimir on Kubernetes. As part of the Mimir 2.3 release, we’re also releasing version 3.1 of the Mimir Helm chart.

Notable enhancements follow. For the full list of changes, see the Helm chart changelog.

We’ve upgraded the MinIO subchart dependency from a deprecated chart to the supported one. This creates a breaking change in how the administrator password is set. However, as the built-in MinIO is not a recommended object store for production use cases, this change did not warrant a new major version of the Mimir Helm chart.
Query sharding is now enabled by default which should give you better performance on high cardinality metrics queries.
- To compensate for the increased number of queries generated by query sharding, the query scheduler component is now enabled by default.
The backfill API endpoints for importing historic time series data are now exposed on the Nginx gateway.
Nginx now sets the value of the X-Scope-OrgID header equal to the value of Mimir’s no_auth_tenant parameter by default. The previous release had set the value of X-Scope-OrgID to anonymous by default which complicated the process of migrating to Mimir.
Memberlist now uses DNS service-discovery by default, which decreases startup time for large Mimir clusters.

Important changes

In Grafana Mimir 2.3 we have removed the following previously deprecated configuration options:

The extend_writes parameter in the distributor YAML configuration and -distributor.extend-writes CLI flag have been removed.
The active_series_custom_trackers parameter has been removed from the YAML configuration. It had already been moved to the runtime configuration. See #1188 for details.
The blocks-storage.tsdb.isolation-enabled parameter in the YAML configuration and -blocks-storage.tsdb.isolation-enabled CLI flag have been removed.

With Grafana Mimir 2.3 we have also updated the default value for the CLI flag -distributor.ha-tracker.max-clusters to 100 to provide Denial-of-Service protection. Previously -distributor.ha-tracker.max-clusters was unlimited by default which could allow a tenant with HA Dedupe enabled to overload the HA tracker with __cluster__ label values that could cause the HA Dedupe database to fail.

Also, as noted above, the administrator password for Helm chart deployments using the built-in MinIO is now set differently.

Bug fixes

PR 2447: Fix incorrect mapping of http status codes 429 to 500 when the request queue is full in the query-frontend. This corrects behavior in the query-frontend where a retryable 429 "Too Many Outstanding Requests" error from a querier was incorrectly returned as an unretryable 500 system error.
PR 2505: The Memberlist key-value (KV) store now tries to “fast-join” the cluster to avoid serving an empty KV store. This fix addresses the confusing “empty ring” error response and the error log message “ring doesn’t exist in KV store yet” emitted by services when there are other members present in the ring when a service starts. Those using other key-value store options (e.g., consul, etcd) are not impacted by this bug.
PR 2289: The “List Prometheus rules” API endpoint of the Mimir Ruler component is no longer blocked while rules are being synced. This means users can now list rules while syncing larger rule sets.

Grafana Mimir version 2.4 release notes

Mon, 17 Mar 2025 08:45:08 +0100

Grafana Mimir version 2.4 release notes

Grafana Labs is excited to announce version 2.4 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Note: If you are upgrading from Grafana Mimir 2.3, review the list of important changes that follow.

Features and enhancements

Query-scheduler ring-based service discovery: The query-scheduler is an optional, stateless component that retains a queue of queries to execute, and distributes the workload to available queriers. The use the query-scheduler, query-frontends and queriers are required to discover the addresses of the query-scheduler instances.

In addition to DNS-based service discovery, Mimir 2.4 introduces the ring-based service discovery for the query-scheduler. When enabled, the query-schedulers join their own hash ring (similar to other Mimir components), and the query-frontends and queriers discover query-scheduler instances via the ring.

Ring-based service discovery makes it easier to set up the query-scheduler in environments where you can’t easily define a DNS entry that resolves to the running query-scheduler instances. For more information, refer to query-scheduler configuration.
New API endpoint exposes per-tenant limits: Mimir 2.4 introduces a new API endpoint, which is available on all Mimir components that load the runtime configuration. The endpoint exposes the limits of the authenticated tenant. You can use this new API endpoint when developing custom integrations with Mimir that require looking up the actual limits that are applied on a given tenant. For more information, refer to Get tenant limits.
New TLS configuration options: Mimir 2.4 introduces new options to configure the accepted TLS cipher suites, and the minimum versions for the HTTP and gRPC clients that are used between Mimir components, or by Mimir to communicate to external services such as Consul or etcd.

You can use these new configuration options to override the default TLS settings and meet your security policy requirements. For more information, refer to Securing Grafana Mimir communications with TLS.
Maximum range query length limit: Mimir 2.4 introduces the new configuration option -query-frontend.max-total-query-length to limit the maximum range query length, which is computed as the query’s end minus start timestamp. This limit is enforced in the query-frontend and defaults to -store.max-query-length if unset.

The new configuration option allows you to set different limits between the received query maximum length (-query-frontend.max-total-query-length) and the maximum length of partial queries after splitting and sharding (-store.max-query-length).

The following experimental features have been promoted to stable:

Helm chart improvements

The mimir-distributed Helm chart is the best way to install Mimir on Kubernetes. As part of the Mimir 2.4 release, we’re also releasing version 3.2 of the mimir-distributed Helm chart.

Notable enhancements follow. For the full list of changes, see the Helm chart changelog.

Added support for topologySpreadContraints.
Replaced the default anti-affinity rules with topologySpreadContraints for all components which puts less restrictions on where Kubernetes can run pods.

Important: if you are not using the sizing plans (small.yaml, large.yaml, capped-small.yaml, capped-large.yaml) in production, you must reintroduce pod affinity rules for the ingester and store-gateway. This also fixes a missing label selector for the ingester. Merge the following with your custom values file:

ingester:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - ingester
          topologyKey: "kubernetes.io/hostname"
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: "kubernetes.io/hostname"
store_gateway:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - store-gateway
          topologyKey: "kubernetes.io/hostname"
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - store-gateway
          topologyKey: "kubernetes.io/hostname"

Updated the anti affinity rules in the sizing plans (small.yaml, large.yaml, capped-small.yaml, capped-large.yaml). The sizing plans now enforce that no two pods of the ingester, store-gateway, or alertmanager StatefulSets are scheduled on the same Node. Pods from different StatefulSets can share a Node.
Support for Openshift Route resource for nginx has been added.

Important changes

In Grafana Mimir 2.4, the default values of the following configuration options have changed:

-distributor.remote-timeout has changed from 20s to 2s.
-distributor.forwarding.request-timeout has changed from 10s to 2s.
-blocks-storage.tsdb.head-compaction-concurrency has changed from 5 to 1.
The hash-ring heartbeat period for distributors, ingesters, rulers, and compactors has increased from 5s to 15s.

In Grafana Mimir 2.4, the following deprecated configuration options have been removed:

The YAML configuration option limits.active_series_custom_trackers_config.
The CLI flag -ingester.ring.join-after and its respective YAML configuration option ingester.ring.join_after.
The CLI flag -querier.shuffle-sharding-ingesters-lookback-period and its respective YAML configuration option querier.shuffle_sharding_ingesters_lookback_period.

With Grafana Mimir 2.4, the anonymous usage statistics tracking is enabled by default. Mimir maintainers use this anonymous information to learn more about how the open source community runs Mimir and what the Mimir team should focus on when working on the next features and documentation improvements. If possible, we ask you to keep the usage reporting feature enabled. In case you want to opt-out from anonymous usage statistics reporting, refer to Disable the anonymous usage statistics reporting.

Bug fixes

PR 2979: Fix remote write HTTP response status code returned by Mimir when failing to write only to one ingester (the quorum is still honored when running Mimir with the default replication factor of 3) and some series are not ingested because of validation errors or some limits being reached.
PR 3005: Fix the querier to re-balance its workers connections when a query-frontend or query-scheduler instance is terminated.
PR 2963: Fix the remote read endpoint to correctly support the Accept-Encoding: snappy HTTP request header.

Grafana Mimir version 2.5 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.5 release notes

Grafana Labs is excited to announce version 2.5 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Alertmanager Discord support Alertmanager can now be configured to send alerts in Discord channels.
Configurable TLS minimum version and cipher suites We added the flags -server.tls-min-version and -server.tls-cipher-suites that can be used to define the minimum TLS version and the supported cipher suites in all HTTP and gRPC servers in Mimir.
Lower memory usage in store-gateway, ingester and alertmanager We made various changes related to how index lookups are performed and how the active series custom trackers are implemented, which results in better performance and lower overall memory usage in the store-gateway and ingester. We also optimized the alertmanager, which results in a 50% reduction in memory usage in use cases with larger numbers of tenants.
Improved Mimir dashboards We added two new dashboards named Mimir / Overview resources and Mimir / Overview networking. Furthermore, we have made various improvements to the following existing dashboards:
- Mimir / Overview: Add “remote read”, “metadata”, and “exemplar” queries.
- Mimir / Writes: Add optional row about the distributor’s new forwarding feature.
- Mimir / Tenants: Add insights into the read path.

Helm chart improvements

Zone aware replication Helm now supports deploying the ingesters and store-gateways as different availability zones. The replication is also zone-aware, therefore multiple instances of one zone can fail without any service interruption and roll outs can be performed faster because many instances of each zone can be restarted together, as opposed to them all restarting in sequence.

This is a breaking change, for details on how to upgrade please review the Helm changelog.
Running without root privileges All Mimir, GEM and Agent processes now don’t require root privileges to run anymore.
Unified reverse proxy (gateway) configuration for Mimir and GEM This change allows for an easier upgrade path from Mimir to GEM, without any downtime. The unified configuration also makes it possible to autoscale the GEM gateway pods and it supports OpenShift Route. The change also deprecates the nginx section in the configuration. The section will be removed in release 7.0.0.
Updated MinIO The MinIO sub-chart was updated from 4.x to 5.0.0, note that this update inherits a breaking change because the MinIO gateway mode was removed.
Updated sizing plans We updated our sizing plans to make them reflect better how we recommend running Mimir and GEM in production. Note that this includes a breaking change for users of the “small” plan, more details can be found in the Helm changelog.
Various quality of life improvements
- Rollout strategies without downtime
- Read path and compactor configuration refresh, providing better default settings
- OTLP ingestion support in the Nginx configuration
- A default configuration for alertmanager, so the user interface and the sending of alerts from the ruler works out of the box

Bug fixes

Flusher: Added Overrides as a dependency to prevent panics when starting with -target=flusher. PR 3151
Query-frontend: properly close gRPC streams to the query-scheduler to stop memory and goroutines leak. PR 3302
Ruler: persist evaluation delay configured in the rulegroup. PR 3392
Fix panics in OTLP ingest path when parse errors occur. PR 3538

Grafana Mimir version 2.6 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.6 release notes

Grafana Labs is excited to announce version 2.6 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Lower memory usage in store-gateway by streaming series results The store-gateway can now stream results back to the querier instead of buffering them. This is expected to greatly reduce peak memory consumption while keeping latency the same. This is still an experimental feature but Grafana Labs is already running it in production and there’s no known issue. This feature can be enabled setting the -blocks-storage.bucket-store.batch-series-size configuration option (if you want to try it out, we recommend you setting to 5000).
Improved stability in store-gateway by removing mmap usage The store-gateway can now use an alternate code path to read index-headers that does not use memory mapped files. This is expected to improve stability of the store-gateway. This is still an experimental feature but Grafana Labs is already running it in production and there’s no known issue. This feature can be enabled setting -blocks-storage.bucket-store.index-header.stream-reader-enabled=true.

Alertmanager improvements

Webex support Alertmanager can now use Webex to send alerts.
tenantID template function A new template function tenantID, returning the ID of the tenant owning the alert, has been added.
grafanaExploreURL template function A new template function grafanaExploreURL, returning the URL to the Grafana explore page with range query, has been added.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the corresponding documentation for more information.

Important changes

In Grafana Mimir 2.6 we have removed the following previously deprecated or experimental configuration options:

The CLI flag -blocks-storage.bucket-store.max-concurrent-reject-over-limit and its respective YAML configuration option blocks_storage.bucket_store.max_concurrent_reject_over_limit.
The CLI flag -query-frontend.align-querier-with-step and its respective YAML configuration option frontend.align_querier_with_step.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.8:

The CLI flag -store.max-query-length and its respective YAML configuration option limits.max_query_length have been replaced with -querier.max-partial-query-length and limits.max_partial_query_length.

The following experimental options and features are now stable:

The CLI flag -query-frontend.max-total-query-length and its respective YAML configuration option limits.max_total_query_length.
The CLI flags -distributor.request-rate-limit and -distributor.request-burst-limit and their respective YAML configuration options limits.request_rate_limit and limits.request_rate_burst.
The CLI flag -ingester.max-global-exemplars-per-user and its respective YAML configuration option limits.max_global_exemplars_per_user.
The CLI flag -ingester.tsdb-config-update-period its respective YAML configuration option ingester.tsdb_config_update_period.
The API endpoint /api/v1/query_exemplars.

Bug fixes

Alertmanager: Fix template spurious deletion with relative data dir. PR 3604
Security: Update prometheus/exporter-toolkit for CVE-2022-46146. PR 3675
Security: Update golang.org/x/net for CVE-2022-41717. PR 3755
Debian package: Fix post-install, environment file path and user creation. PR 3720
Memberlist: Fix panic during Mimir startup when Mimir receives gossip message before it’s ready. PR 3746
Update github.com/thanos-io/objstore to address issue with Multipart PUT on s3-compatible Object Storage. PR 3802 PR 3821
Querier: Canceled requests are no longer reported as “consistency check” failures. PR 3837 PR 3927
Distributor: Don’t panic when metric_relabel_configs in overrides contains null element. PR 3868
Ingester, Compactor: Fix panic that can occur when compaction fails. PR 3955

2.6.1

Security: updated the Go version to 1.20.3 to fix CVE-2023-24538. PR 4798

2.6.2

Security: updated Go to version 1.20.4 to fix CVE-2023-24539, CVE-2023-24540, CVE-2023-29400. PR 4903

Grafana Mimir version 2.7 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.7 release notes

Grafana Labs is excited to announce version 2.7 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Store-gateway streaming enabled by default The new default value of 5000 for -blocks-storage.bucket-store.batch-series-size enables store-gateway streaming in the default configuration. This means that series are loaded from object storage in batches rather than buffering them all in memory before returning to the querier. Enabling streaming can reduce memory utilization peaks in the store-gateway.
Store-gateway index header reader no longer uses mmap by default Along with streaming enabled in the store-gateway, this change contributes to more efficient memory usage. See the Important changes section for more details.
Support for keep_firing_for option to ruler configuration This new option determines the amount of time an alert should keep firing while the ruler expression doesn’t return results.
More efficient chunks fetching and caching Enable with the new experimental feature flag -blocks-storage.bucket-store.chunks-cache.fine-grained-chunks-caching-enabled=true. This should reduce CPU, memory utilization, and receive bandwidth of a store-gateway.
Experimental query sharding improvements: A new configuration parameter, -query-frontend.query-sharding-target-series-per-shard, allows query sharding to take into account cardinality of similar requests executed previously when computing the maximum number of shards to use. If you want to try it out, we recommend starting with a value of 2500.
Experimental support for native histogram ingestion: Native histograms can now be ingested. The new per-tenant limit -ingester.native-histograms-ingestion-enabled controls whether native histograms are stored or ignored. The support for querying native histograms is not complete yet and it’s expected to be available in the next release.

Alertmanager improvements

New metrics The following upstream metrics are now exposed:
- cortex_alertmanager_dispatcher_aggregation_groups
- cortex_alertmanager_dispatcher_alert_processing_duration_seconds

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.7, the default vaules of the following configuration options have changed:

-blocks-storage.bucket-store.batch-series-size is now enabled by default with a value of 5000.
-ruler.evaluation-delay-duration has changed from 0 to 1m.

In Grafana Mimir 2.7, the following configuration options are now deprecated:

-blocks-storage.bucket-store.chunks-cache.subrange-size since there’s no benefit to changing the default of 16000
-blocks-storage.bucket-store.consistency-delay has been deprecated and will be removed in Mimir 2.9.
-compactor.consistency-delay has been deprecated and will be removed in Mimir 2.9.
-ingester.ring.readiness-check-ring-health has been deprecated and will be removed in Mimir 2.9.

In Grafana Mimir 2.7, the following options, metrics, and labels have been removed:

Experimental support for ephemeral storage introduced in Mimir 2.6.0 has been removed.
- Following options are no longer available:
  - -blocks-storage.ephemeral-tsdb.*
  - -distributor.ephemeral-series-enabled
  - -distributor.ephemeral-series-matchers
  - -ingester.max-ephemeral-series-per-user
  - -ingester.instance-limits.max-ephemeral-series
- The following metrics have been removed:
  - cortex_ingester_ephemeral_series
  - cortex_ingester_ephemeral_series_created_total
  - cortex_ingester_ephemeral_series_removed_total
  - cortex_ingester_ingested_ephemeral_samples_total
  - cortex_ingester_ingested_ephemeral_samples_failures_total
  - cortex_ingester_memory_ephemeral_users
  - cortex_ingester_queries_ephemeral_total
  - cortex_ingester_queried_ephemeral_samples
  - cortex_ingester_queried_ephemeral_series
- Additionally, querying using the {__mimir_storage__="ephemeral"} selector no longer works. All label values with the ephemeral- prefix within the reason label of the cortex_discarded_samples_total metric are no longer available.
The store-gateway default index header reader no longer uses mmap and the mmap-based index header reader has been removed. The following flags have been changed:
- -blocks-storage.bucket-store.index-header.map-populate-enabled has been removed
- -blocks-storage.bucket-store.index-header.stream-reader-enabled has been removed
- -blocks-storage.bucket-store.index-header.stream-reader-max-idle-file-handles has been renamed to -blocks-storage.bucket-store.index-header.max-idle-file-handles, and the corresponding configuration file option has been renamed from stream_reader_max_idle_file_handles to max_idle_file_handles

Bug fixes

Store-gateway: return Canceled rather than Aborted or Internal error when the calling querier cancels a label names or values request, and return Internal if processing the request fails for another reason. PR 4061
Querier: track canceled requests with status code 499 in the metrics instead of 503 or 422. PR 4099
Ingester: compact out-of-order data during /ingester/flush or when TSDB is idle. PR 4180
Ingester: conversion of global limits max-series-per-user, max-series-per-metric, max-metadata-per-user and max-metadata-per-metric into corresponding local limits now takes into account the number of ingesters in each zone. PR 4238
Ingester: track cortex_ingester_memory_series metric consistently with cortex_ingester_memory_series_created_total and cortex_ingester_memory_series_removed_total. PR 4312
Querier: fixed a bug which was incorrectly matching series with regular expression label matchers with begin/end anchors in the middle of the regular expression. PR 4340
Security: updated the Go version to 1.20.3 to fix CVE-2023-24538. PR 4795
Security: updated the Go version to 1.20.4 to fix CVE-2023-24539, CVE-2023-24540, CVE-2023-29400. PR 4905

Grafana Mimir version 2.8 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.8 release notes

Grafana Labs is excited to announce version 2.8 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Experimental support for using Redis as cache Mimir now can use Redis for caching results, chunks, index and metadata.
Experimental support for fetching secret from Vault for TLS configuration.
Experimental support for querying native histograms. This support is not finalized as the related Prometheus API is also experimental, thus the exact behavior might change in future releases.
Query-frontend and ruler now use protobuf internal query result payload format by default This reduces the CPU and memory utilisation of the querier, query-frontend and ruler, as well as reducing network bandwidth consumed between these components.
Query-frontend cached results now contain timestamp This allows Mimir to check if cached results are still valid based on current TTL configured for tenant. Results cached by previous Mimir version are used until they expire from cache, which can take up to 7 days. If you need to use per-tenant TTL sooner, please flush results cache manually.
Optimized regular expression label matchers This reduces CPU utilisation in ingesters and store-gateways when running queries containing regular expression label matchers.
Store-gateway now use streaming for LabelNames RPC This improves memory utilization in store-gateway when calling LabelNames RPC.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.8 we have removed the following previously deprecated or experimental metrics:

cortex_bucket_store_series_get_all_duration_seconds
cortex_bucket_store_series_merge_duration_seconds
cortex_ingester_tsdb_wal_replay_duration_seconds

The following configuration options are deprecated and will be removed in Grafana Mimir 2.10:

The CLI flag -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup and its respective YAML configuration option tsdb.max_tsdb_opening_concurrency_on_startup.

The following configuration options that were deprecated in 2.6 are removed:

The CLI flag -store.max-query-length and its respective YAML configuration option limits.max_query_length.

The following configuration options that were deprecated in 2.5 are removed:

The CLI flag -azure.msi-resource.

The following experimental options and features are now stable:

The protobuf internal query result payload format, which is now enabled by default

We changed default value of block storage retention period. The default value for -blocks-storage.tsdb.retention-period was 24h and now is 13h

Bug fixes

Querier: Streaming remote read will now continue to return multiple chunks per frame after the first frame. PR 4423
Query-frontend: don’t retry queries which error inside PromQL. PR 4643
Store-gateway & query-frontend: report more consistent statistics for fetched index bytes. PR 4671

Grafana Mimir version 2.9 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.9 release notes

Grafana Labs is excited to announce version 2.9 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Reduced store-gateway memory utilization on fetching series from long-term storage For queries that include broad label matchers (e.g. datacenter="dc1"), Mimir 2.9 will fetch a reduced volume of index data, which leads to a significant reduction in memory allocations in the store-gateway.
Reduced CPU utilisation for some shuffle sharding scenarios Mimir queriers will now use significantly less CPU in cases where shuffle sharding is enabled for tenants with a shard size that’s large but lower than the total number of ingesters.
Reduced object storage API calls in compactors and rulers Mimir 2.9 comes with optimizations that will reduce the amount of times compactors and rulers need to access rules stored in object storage.
- This release adds experimental support for a ruler storage cache. This cache should reduce the number of “list objects” API calls issued to the object storage when there are 2+ ruler replicas running in a Mimir cluster. The cache can be configured by setting the -ruler-storage.cache.* CLI flags or their respective YAML config options.
- We also introduced a new feature to trigger a synchronization of tenant’s rule groups as soon as changes to the rule configuration are made via API. This synchronization is in addition of the periodic syncing done every -ruler.poll-interval, which has then been relaxed from every 1m to every 10m. The new behaviour is enabled globally by default but can be disabled with -ruler.sync-rules-on-changes-enabled=false or tuned at a per-tenant level.
Experimental support for streaming chunks from ingester to querier This is expected to greatly reduce querier memory consumption when evaluating queries that select a large number of series, because chunks streamed from the querier can now be read into memory as needed.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.9 we have removed the following previously deprecated or experimental metrics:

cortex_bucket_store_chunk_pool_requested_bytes_total
cortex_bucket_store_chunk_pool_returned_bytes_total

The following configuration options are deprecated and will be removed in Grafana Mimir 2.11:

The CLI flag -querier.query-ingesters-within. This configuration is moved to per-tenant overrides.
The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
The CLI flags -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes, -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes and -blocks-storage.bucket-store.max-chunk-pool-bytes.
The CLI flags -querier.iterators and -query.batch-iterators.

The following configuration options that were deprecated in 2.7 are removed:

The CLI flag -blocks-storage.bucket-store.chunks-cache.subrange-size. A fixed value of 16000 is now always used.
The CLI flag -blocks-storage.bucket-store.consistency-delay.
The CLI flag -compactor.consistency-delay.
The CLI flag -ingester.ring.readiness-check-ring-health.

The following experimental options and features are now stable:

The CLI flag -query-frontend.query-sharding-max-regexp-size-bytes.
The CLI flag -query-scheduler.max-used-instances.
The CLI flags -(alertmanager|blocks|ruler)-storage.storage-prefix.
The CLI flag -compactor.first-level-compaction-wait-period.
The CLI flags -usage-stats.enabled and -usage-stats.installation-mode.
The CLI flag -query-frontend.query-sharding-target-series-per-shard.

The following configuration option defaults were changed:

The default value for the CLI flag -query-frontend.query-sharding-max-regexp-size-bytes was changed from 0 to 4096. As a result, queries with regex matchers exceeding this limit will not be sharded by default.
The default value for the CLI flag -compactor.partial-block-deletion-delay was changed from 0s to 1d. As a result, partial blocks resulting from a failed block upload or deletion will be cleaned up automatically.
The default value for the CLI flag -ruler.poll-interval was changed from 1m to 10m.

Bug fixes

Store-gateway: Detect collisions in the postings cache. PR 4770
Store-gateway: Fix panic caused by cached LabelValues responses with more than 655360 values. PR 5021

2.9.4

Security: updated the Alpine base image version to 3.18.5 to address CVE-2023-5363. PR 6895

Grafana Mimir version 2.10 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.10 release notes

Grafana Labs is excited to announce version 2.10 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Added support for rule filtering by passing file, ruler_group and rule_name parameters to the ruler endpoint /api/v1/rules.
Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_values by passing the count_method parameter. You can set it to active to count only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.
Reduced the overall memory consumption by changing the internal data structure for labels. Expect ingesters to use around 15% less memory with this change, depending on the pattern of labels used, number of tenants, etc.
Reduced the memory usage of the Active Series Tracker in the ingester.
Added a buffered logging implementation that can be enabled through the -log.buffered CLI flag. This should reduce contention and resource usage under heavy usage patterns.
Improved the performance of the OTLP ingestion and more detailed information was added to the traces in order to make troubleshooting problems easier.
Improved the performance of series matching in the store-gateway by always including the __name__ posting group causing a reduction in the number of object storage API calls.
Improved the performance of label values with matchers calls when number of matched series is small. If you’re using Grafana to query Grafana Mimir, you’ll need to be sure your Prometheus data source configuration has the Prometheus type set to Mimir and the Version set correctly in order to benefit from this improvement.
Support to cache cardinality, label names and label values query responses in query frontend. The cache will be used when -query-frontend.cache-results is enabled, and -query-frontend.results-cache-ttl-for-cardinality-query or -query-frontend.results-cache-ttl-for-labels-query is set to a value greater than 0.
Reduced wasted effort spent computing results that won’t be used by having queriers cancel the requests sent to the ingesters in a zone upon receiving first error from that zone.
Reduced object storage use by enhancing the compactor to remove the bucket index, markers, and debug files when it detects zero remaining blocks in the bucket index. This cleanup process can be enabled by setting the -compactor.no-blocks-file-cleanup-enabled option to true.
Added new debug HTTP endpoints /ingester/tenants and /ingester/tsdb/{tenant} to the ingester that provide debug information about tenants and their TSDBs.
Added new metrics for tracking native histograms in active series: cortex_ingester_active_native_histogram_series, cortex_ingester_active_native_histogram_series_custom_tracker, cortex_ingester_active_native_histogram_buckets, cortex_ingester_active_native_histogram_buckets_custom_tracker. The first 2 are the subsets of the existing and unmodified cortex_ingester_active_series and cortex_ingester_active_series_custom_tracker respectively, only tracking native histogram series, and the last 2 are the equivalent for tracking the number of buckets in native histogram series.

Additionally, the following previously experimental features are now considered stable:

Support for a ruler storage cache. This cache should reduce the number of “list objects” API calls issued to the object storage when there are 2+ ruler replicas running in a Mimir cluster. The cache can be configured by setting the -ruler-storage.cache.* CLI flags or their respective YAML config options.
Query sharding cardinality estimation. This feature allows query sharding to take into account cardinality of similar requests executed previously when computing the maximum number of shards to use. You can enable it through the advanced CLI configuration flag -query-frontend.query-sharding-target-series-per-shard; we recommend starting with a value of 2500.
Query expression size limit. You can limit the size in bytes of the queries allowed to be processed through the CLI configuration flag -query-frontend.max-query-expression-size-bytes.
Peer discovery / tenant sharding for overrides exporters. You can enable it through the CLI configuration flag -overrides-exporter.ring.enabled.
Overrides exporter enabled metrics selection. You can select which metrics the overrides exporter should export through the CLI configuration flag -overrides-exporter.enabled-metrics.
Per-tenant results cache TTL. The time-to-live duration for cached query results can be configured using the results_cache_ttl and results_cache_ttl_for_out_of_order_time_window parameters.

Experimental features

Grafana Mimir 2.10 includes new features that are considered as experimental and disabled by default. Please use them with caution and report any issues you encounter:

Support for ingesting exponential histograms in OpenTelemetry format. The exponential histograms that are over the native histogram scale limit of 8 are downscaled to allow their ingestion.
Store-gateway index-header loading improvements, which include the ability to persist the sparse index-header to disk instead of reconstructing it on every restart (-blocks-storage.bucket-store.index-header-sparse-persistence-enabled) as well as the ability to persist the list of block IDs that were lazy-loaded while running to eagerly load them upon startup to prevent starting up with no loaded blocks (-blocks-storage.bucket-store.index-header.eager-loading-startup-enabled) and an option to limit the number of concurrent index-header loads when lazy-loading (-blocks-storage.bucket-store.index-header-lazy-loading-concurrency).
Option to allow queriers to reduce pressure on ingesters by initially querying only the minimum set of ingesters required to reach quorum. (-querier.minimize-ingester-requests).
Early TSDB Head compaction in the ingesters to reduce in-memory series when a certain threshold is reached. Useful to deal with high series churning rate. (-blocks-storage.tsdb.early-head-compaction-min-in-memory-series).
Spread-minimizing token generation algorithm for the ingesters. This new method drastically reduces the difference in series pushed to different ingesters. Please note that a migration process is required to switch from previous random generation algorithm, which will be detailed once the feature is declared stable.
Support for chunks streaming from store-gateways to queriers that should reduce the memory usage in the queriers. Can be enabled through the -querier.prefer-streaming-chunks-from-store-gateways option.
Support for circuit-breaking the distributor write requests to the ingesters. This can be enabled through the -ingester.client.circuit-breaker.* configuration options and should serve to let ingesters recover when under high pressure.
Support to limit read requests based on CPU/memory utilization. This should alleviate pressure on the ingesters after receiving heavy queries and reduce the likelihood of disrupting the write path. (-ingester.read-path-cpu-utilization-limit, -ingester.read-path-memory-utilization-limit, -ingester.log-utilization-based-limiter-cpu-samples).

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.10 we have changed the following behaviors:

Query requests are initiated only to ingesters in the ACTIVE state in the ring. This is not expected to introduce any degradation in terms of query results correctness or high-availability.
Per-instance limit errors are not logged anymore, to reduce resource usage when ingesters are under pressure. We encourage you to use metrics and alerting to monitor them instead. The following metrics have been added to count the number of requests rejected for hitting per-instance limits:
- cortex_distributor_instance_rejected_requests_total
- cortex_ingester_instance_rejected_requests_total
The CLI flag -validation.create-grace-period is now enforced in the ingester. If you’ve configured -validation.create-grace-period, make sure the configuration is applied to ingesters too.
The CLI flag -validation.create-grace-period is now enforced for exemplars. The cortex_discarded_exemplars_total{reason="exemplar_too_far_in_future",user="..."} series is incremented when exemplars are dropped because their timestamp is greater than “now + grace_period”.
The CLI flag -validation.create-grace-period is now enforced in the query-frontend even when the configured value is 0. When the value is 0, the query end time range is truncated to the current real-world time.

The following metrics were removed:

cortex_ingester_shipper_dir_syncs_total
cortex_ingester_shipper_dir_sync_failures_total

The following configuration options are deprecated and will be removed in Grafana Mimir 2.12:

The CLI flag -blocks-storage.bucket-store.index-header-lazy-loading-enabled is deprecated, use the new configuration -blocks-storage.bucket-store.index-header.lazy-loading-enabled.
The CLI flag -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout is deprecated, use the new configuration -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout.
The CLI flag -blocks-storage.bucket-store.index-header-lazy-loading-concurrency is deprecated, use the new configuration -blocks-storage.bucket-store.index-header.lazy-loading-concurrency.

The following configuration options that were deprecated in Grafana Mimir 2.8 are removed:

The CLI flag blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup.

The following experimental configuration options were renamed or removed:

The CLI flag -querier.prefer-streaming-chunks was renamed to -querier.prefer-streaming-chunks-from-ingesters.
The CLI flag -blocks-storage.bucket-store.chunks-cache.fine-grained-chunks-caching-enabled was removed.
The CLI flag -blocks-storage.bucket-store.fine-grained-chunks-caching-ranges-per-series was removed.

The following experimental options are now stable:

The CLI flag -shutdown-delay.
The CLI flag -ingester.ring.excluded-zones.

The following configuration option defaults were changed:

The default value for the CLI flag -querier.streaming-chunks-per-ingester-buffer-size was changed from 512 to 256.
The default value for gRPC clients connect timeout was set to 5s (default inherited from gRPC client was 20s) with a default max backoff delay of 5s (default inherited from gRPC client was 120s).

Bug fixes

Ruler: fixed graceful shutdown for rule evaluations.
Ingester: fixed ingesters getting stuck when previous state is LEAVING and the number of tokens has changed upon restarting.
Querier: fixed timestamp() function fail with execution: attempted to read series at index 0 from stream, but the stream has already been exhausted if the experimental feature to stream chunks from ingesters to queriers is enabled.
Memberlist: brought back memberlist_client_kv_store_count metric that used to exist in Cortex, but got lost during grafana/dskit updates before Mimir 2.0.
Store-gateway: fixed an issue where stopping a store-gateway could cause all store-gateways to unload all blocks.
Ingester: prevented setting “last update time” of TSDB into the future when opening TSDB. This could prevent detecting of idle TSDB for a long time, if sample in distant future was ingested.
General: changed ballast to allocate smaller blocks to avoid problem when entire ballast was kept in memory working set.

2.10.5

Security: updated the Alpine base image version to 3.18.5 to address CVE-2023-5363. PR 6897

Grafana Mimir version 2.11 release notes

Thu, 21 Dec 2023 22:21:36 +0000

Grafana Mimir version 2.11 release notes

Grafana Labs is excited to announce version 2.11 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Sampled logging of errors in the ingester. A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the -ingester.error-sample-rate CLI flag.
Add total request size instance limit for ingesters. This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the -ingester.instance-limits.max-inflight-push-requests-bytes CLI flag in combination with the -ingester.limit-inflight-requests-using-grpc-method-limiter CLI flag.
Reduce the resolution of incoming native histograms samples if the incoming sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default but can be turned off by setting the -validation.reduce-native-histogram-over-max-buckets CLI flag to false.
Improved query-scheduler performance under load. This is particularly apparent for clusters with large numbers of queriers.
Ingester to querier chunks streaming reduces the memory utilization of queriers and reduces the likelihood of OOMs.
Ingester query request minimization reduces the number of query requests to ingesters, improving performance and resource utilization for both ingesters and queriers.
The markblocks tool is released in compiled form together with the rest of Mimir. It is used to mark blocks for deletion or as “non-compactable”.

Experimental features

Grafana Mimir 2.11 includes new features that are considered experimental and disabled by default. Please use them with caution and report any issue you encounter:

Block specified queries on a per-tenant basis. This is configured via the blocked_queries limit. See the docs for more information.
Store metadata when ingesting metrics via OTLP. This makes metric description and type available when ingesting metrics via OTLP. You can enable this feature by setting the CLI flag -distributor.enable-otlp-metadata-storage to true.
Reject gRPC push requests that the ingester/distributor is unable to accept before reading them into memory. You can enable this feature by using the -ingester.limit-inflight-requests-using-grpc-method-limiter and/or the -distributor.limit-inflight-requests-using-grpc-method-limiter CLI flags for the ingester and/or the distributor, respectively.
Customize the memcached client write and read buffer size. The buffer allocated for each memcached connection can be configured via the following CLI flags:
- For the blocks storage:
  - -blocks-storage.bucket-store.chunks-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.chunks-cache.memcached.write-buffer-size-bytes
  - -blocks-storage.bucket-store.index-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.index-cache.memcached.write-buffer-size-bytes
  - -blocks-storage.bucket-store.metadata-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.metadata-cache.memcached.write-buffer-size-bytes
- For the query frontend:
  - -query-frontend.results-cache.memcached.read-buffer-size-bytes
  - -query-frontend.results-cache.memcached.write-buffer-size-bytes
- For the ruler storage:
  - -ruler-storage.cache.memcached.read-buffer-size-bytes
  - -ruler-storage.cache.memcached.write-buffer-size-bytes
Configure the number of long-living workers used to process gRPC requests. This can decrease CPU usage by reducing the number of stack allocations. Configure this feature by using the -server.grpc.num-workers CLI flag.
Enforce a limit in bytes on the PostingsForMatchers cache used by ingesters. This limit can be configured via the -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes CLI flags.
Pre-allocate the pool of workers in the distributor that are used to send push requests to ingesters. This can decrease CPU usage by reducing the number of stack allocations. You can enable this feature by using the -distributor.reusable-ingester-push-worker flag.
Include a Retry-After header in recoverable error responses from the distributor. This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the -distributor.retry-after-header.enabled CLI flag.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.11 the following behavior has changed:

The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Mimir 2.11:

The CLI flag -querier.iterators.
The CLI flag -query.batch-iterators.
The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.13:

The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

Bug fixes

Ingester: Respect context cancelation during query execution. PR 6085
Distributor: Return 529 when ingestion rate limit is hit and the distributor.service_overload_status_code_on_rate_limit_enabled flag is active. PR 6549
Query-scheduler: Prevent accumulation of stale querier connections. PR 6100
Packaging: Fix preremove script preventing upgrades on RHEL based OS. PR 6067
Ingester: Fixed possible series matcher corruption leading to wrong series being included in query results. PR 6884
Mimirtool: Fixed missing .metricsUsed field in the output of analyze rule-file. PR 6953