Release notes on Grafana Labs

Grafana Enterprise Metrics v1.6 release notes

Fri, 03 Apr 2026 19:43:06 +0000

Version 1.6 release notes

The Grafana Enterprise Metrics (GEM) team is excited to announce the release of GEM 1.6.

Features and enhancements

GEM 1.6 adds 2 new API endpoints to help users explore the metrics cardinality of tenants within their cluster. These endpoints make it possible to identify the highest cardinality metrics in a tenant, as well as the label names with the highest value count. The GEM plug-in (starting with version 3.3.0) includes a set of cardinality dashboards built on these endpoints to allow users to explore the cardinality data in a visual way. For more information, see the Cardinality Analysis page.
GEM 1.6 includes experimental support for query sharding, which allows a single query to be split up into groups of smaller series and parallelized. This improves the performance of high cardinality queries which touch a lot of series. This feature is still very much under active development, so take care when testing.
GEM 1.6 includes experimental support for a new compaction strategy known as “split and merge compaction”. This strategy enables users to parallelize the compaction of a large tenant horizontally across multiple machines. It also enables users to overcome the 64GiB block index limit inherent in Prometheus, raising the ceiling on how many time series a single tenant can support. As with query sharding, this feature is still under active development and should be enabled with caution. For more information, see the relevant section in Configuration.
GEM 1.6 eliminates the use of the word “instance,” which had been used as a synonym for “tenant”. Switching to “tenant” allows GEM to more closely align with terminology used in Cortex and avoids confusion caused by the fact that “instance” can have multiple meanings besides “tenant” (e.g., an “instance” of an ingester).
- All references to “instance” in the GEM documentation and GEM Grafana plug-in have been replaced with the word “tenant”.
- We’ve added new v2 API endpoints which specify “tenant” in the request and response bodies. We have deprecated and will eventually delete the original endpoints that use the word ‘instance’. For more detail, see the admin-api page.
GEM 1.6 renames the default auth option to trust. This reflects the fact that the trust option does not actually do any authentication, but simply ‘trusts’ the value of the HTTP header X-Scope-Org-Id to determine what tenant to send a request to. GEM users are highly recommended to use auth type enterprise, not trust.
We’ve made some efficiency improvements, including reduced CPU and memory utilization on the ingesters and queries and when seeing large volumes of out-of-order samples.

Upgrade considerations

Please upgrade your Grafana GEM administrative plug-in to version 3.3.0 in order to get the dashboards associated with the Cardinality Analysis feature.
- To ensure you receive the most up-to-date dashboards, you must disable and then re-enable the plug-in. Therefore, once GEM plug-in version 3.3.0 is installed, navigate to its Plugin configuration page, and hit the Disable plugin button. After Grafana reloads, hit the Enable plugin.
See the note under “Changes” about the auth option being renamed from the default to trust. GEM 1.6 sets the default max number of inflight ingester push requests (-ingester.instance-limits.max-inflight-push-requests) to 30000. We don’t anticipate this impacting the steady state operation of any GEM user; it is in place as a safeguard against clusters getting overwhelmed by a spike in requests.
The option -querier.ingester-streaming has been removed. The querier and ruler now always use streaming method to query ingesters.
GEM 1.6 has removed support for block deletion marks migration (-compactor.block-deletion-marks-migration-enabled). If you’re upgrading to GEM 1.6 from GEM < 1.1.0, then you need to upgrade to GEM 1.5 first and run the migration at least once. Then you can upgrade to 1.6.

Bug fixes

1.6.2

GEM 1.6.2 fixes a bug that caused tenant limits set via the Admin API to be invalidated when upgrading to GEM 1.6.0 or GEM 1.6.1 from previous versions of GEM.

1.6.1

1.6.1 fixes one important bug in the 1.6.0 release:

GEM 1.6.1 fixes an issue with the way default tenant limits were applied when setting tenant limits using the Admin API.

1.6.0

GEM 1.6 changes the way per-tenant limits are stored to avoid breaking changes between version updates.
GEM 1.6 has a bugfix to ensure that the recording rules executed by the __system__ tenant (part of the self-monitoring feature) are properly split across rulers. This should eliminate out-of-order errors on the __system__ tenant.
GEM now enforces that a tenant must exist even when a wildcard access policy is used. Writes to non-existent tenants with a wildcard write access policy will fail. Similarly, if there are any read requests to a non-existent tenant using a wildcard write access policy, they will fail.
If a GEM tenant is deleted but the access policies to that GEM tenant are not, GEM will now fail read and write requests to that tenant.

Grafana Enterprise Metrics v1.7 release notes

Fri, 03 Apr 2026 19:43:06 +0000

Version 1.7 release notes

The Grafana Enterprise Metrics (GEM) team is excited to announce the release of GEM 1.7.

Features and enhancements

GEM’s admin-api is now compatible with Azure Blob storage. Check out the admin_client_config section of the global configuration for more details on how to set this up.
Two enhancements have been made to the federation frontend, a service which allows users to query multiple GEM clusters at once and combine the results. Please see Cluster query federation for more information.
- The federation frontend service will now propagate a request’s bearer token, allowing that token to be used for authentication at its downstream targets.
- The federation frontend service now supports TLS configuration to its downstream targets.

Upgrade considerations

Users can upgrade from GEM 1.6.2 to GEM 1.7 with no additional considerations.

Grafana Enterprise Metrics v2.0 release notes

Fri, 03 Apr 2026 19:43:06 +0000

Version 2.0 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.0. GEM 2.0 is built on top of Grafana Mimir 2.0, which is the most scalable, most performant open source time series database in the world. To learn more about the Grafana Mimir project, see our announcement blog post as well as the Grafana Mimir documentation.

GEM 2.0 inherits all of the highlights of Grafana Mimir 2.0, including easy deployment, native multi-tenancy, high availability, durable long-term metrics storage, and exceptional query performance. GEM includes additional functionality that enterprise users need to operate a time series database at their organization, including simplified tenant management, fine-grained access controls, and out-of-the box self-monitoring and usage data.

GEM 2.0 is a major version upgrade from the previous version of GEM, 1.7, and represents a major improvement in terms of usability and functionality over its predecessor. However, some of these improvements introduce breaking changes for those upgrading from GEM 1.7. We did our best to weigh the tradeoffs between the one-time pain and long-term benefits, such as a simplified configuration experience, and reduced maintenance burden, when deciding which breaking changes to introduce. For more information about upgrading, see Upgrade Considerations.

Although GEM releases have historically been based on an arbitrary commit hash of Cortex, going forward we will base GEM releases off of a named release of Grafana Mimir. Doing so simplifies the process of understanding what changes from Grafana Mimir are present in GEM.

Features and enhancements

The majority of features and enhancements in GEM 2.0 are inherited from Grafana Mimir 2.0. Therefore, start by reading about the features and enhancements in the Grafana Mimir 2.0 release notes, as well as the Grafana Mimir 2.0 documentation.

The following list highlights the features and enhancements in Grafana Enterprise Metrics 2.0:

The following features were released in Grafana Mimir 2.0, but have some GEM 2.0 specific implications:
- GEM 2.0 inherits Grafana Mimir 2.0’s experimental support for cross-tenant alerting and recording rules. GEM 2.0 seamlessly integrates with GEM’s access control model to ensure that recording and alerting rules can only be created and executed after proper authorization. For more information, see Cross tenant alerting and recording rule federation
- In Grafana Mimir 2.0, we open sourced and promoted to stable the split-and-merge compaction algorithm that was initially released as an experimental closed-source feature in GEM 1.6. Both Grafana Mimir 2.0 and GEM 2.0 utilize the split-and-merge compaction strategy by default. The closed-source time-sharding compaction strategy introduced in GEM 1.4 has been removed because it is redundant. The split-and-merge compactor contains the vertical scaling capability the time-sharding algorithm introduced and adds horizontal scaling as well. For more information on how to transition from the time-sharded compactor to the split-and-merge compactor, see the migration guide. To learn more about the new split-and-merge compactor, see Compaction in the Grafana Mimir 2.0 docs.
- In Grafana Mimir 2.0, we open sourced the experimental query-sharding feature that was introduced in GEM 1.6. This feature has been promoted to stable, and new documentation about how to enable and configure it is in the Query sharding section of the Grafana Mimir 2.0 docs.
We’ve released a version 3 of the admin-api that removes the ability to hard delete tenants, access policies, and tokens via the admin-api. Instead, when users no longer want a tenant, access policy, or token, they are expected to soft delete it by marking it “inactive”. Moving to soft deletes allows us to eliminate several race conditions and cache invalidation problems that caused unexpected behavior with hard deletes. For more information about the v3 API, see the Admin-API documentation.
For those running GEM on Kubernetes, we’ve released a new version of the GEM Helm chart alongside our GEM 2.0 release. The chart itself includes multiple changes including most notably that the chart was renamed to mimir-distributed. There will now be 1 Helm chart for installing both OSS Grafana Mimir and Grafana Enterprise Metrics, with users using the enterprise.enabled flag to choose between the two. For more detail, see the Helm chart README.
We’ve made two improvements to GEM’s self-monitoring functionality:
- We added several panels to the Compactor dashboard. They display the time since the last successful compaction run and indicate when too much time has elapsed. This can help flag a malfunctioning or under-resourced compactor.
- GEM can now emit exemplars for its self-monitoring counter and histogram metrics, making it easier to jump from self-monitoring metrics to representative traces. For information on how to enable, see the exemplars section of the self-monitoring documentation.
The per-tenant limit compactor_blocks_retention_period can now be set via the admin-api. For more information, see Setting-per-tenant resource usage limits. That being said, we encourage those using the admin-api to migrate to managing per-tenant resource usage limits via runtime configuration because we intend to deprecate the ability to manage resource limits via admin-api in the future.

Upgrade considerations

GEM 2.0 is a major version upgrade and therefore introduces several breaking changes relative to GEM 1.7. The majority of the breaking changes are concentrated in GEM’s configuration, where we removed a significant number configuration flags (454 to be exact) that were no longer needed. We renamed and/or updated the defaults for numerous other flags whose legacy names or default values no longer made sense. This work allowed us to reduce and simplify GEM’s configuration surface and therefore make it more approachable to users.

To make the GEM 1.7 to GEM 2.0 upgrade as easy as possible, we’ve added functionality to mimirtool for automating the changes users need to make to their configuration.

We’ve paired this tooling with a detailed migration guide that walks through all required changes needed to go from GEM 1.7 to GEM 2.0 in a step-by-step manner. The guide also includes specific instructions on how to perform the upgrade using the Helm chart.

Once you upgrade your GEM backend to version 2.0, we recommend you upgrade your Grafana Enterprise GEM plugin to >= v3.4.0. This is needed to get the improved Compactor panel dashboards mentioned in the “Features and enhancements” section. It also ensures the plugin works with the newest version (v3) of the admin-api.

Bug fixes

v2.0.1

Fixed a bug where authentication using the wildcard (*) username will fail when there are tenants deleted via the v3 admin API endpoints.

v2.0.0

The following lists the most important bugs that are fixed in Grafana Enterprise Metrics 2.0.0:

Fixed a bug that caused GEM to report an incorrect license expiration timestamp in the metric grafana_labs_license_expiry_timestamp if multiple valid licenses existed in local storage and object storage.
Enterprise configuration extensions now appear in GEM’s /config endpoint. Before the fix, only configuration values present in the upstream OSS were returned.
Transient errors while attempting to authenticate user requests to GEM now return an HTTP 500 status code instead of an HTTP 401 code. The 500 code ensures that clients like Prometheus or the Grafana agent will now retry their remote write request rather than dropping it.
Transient errors are no longer cached by GEM, ensuring that GEM does not repeatedly return an HTTP 500 status code to the client due to a one-off failure.
When using a label-based access policy with multiple label selectors, GEM now ensures that label values are filtered properly. Before the bug fix, users with an LBAC policy with multiple label selectors could see label values they did not have access to. Metric names and metrics data were already filtered properly and unaffected by this bug.

For the full list of bug fixes, see the Changelog.

Grafana Enterprise Metrics v2.1 release notes

Fri, 03 Apr 2026 19:43:06 +0000

Version 2.1 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.1. GEM 2.1 is built on top of Grafana Mimir 2.1, which means it inherits all its features, enhancements, and bugfixes. Given this, we recommend you start by reading the Grafana Mimir 2.1 release notes.

Below, we highlight the most important features, enhancements, and bugfixes specific to GEM 2.1. For the full list of changes, see the Changelog.

Features and enhancements

The following list highlights the features and enhancements in Grafana Enterprise Metrics 2.1:

Remote ruler mode for improved rule evaluation performance: This experimental feature is inherited from Grafana Mimir 2.1 and is explained in more detail in the Mimir 2.1 release notes. GEM 2.1 adds an additional configuration parameter, -ruler.query-frontend.auth-token, which is required for configuring the remote ruler with GEM enterprise authentication. See Remote rule evaluation for more information.
Page fault metrics for self-monitoring: GEM now collects and stores metrics tracking the number of page faults and out-of-memory kills it experiences, making it easier to debug performance issues.

Upgrade considerations

We’ve released a new version of our Helm chart in tandem with the GEM 2.1 release. The upgrade for users of the Helm chart has no breaking changes.

Once you upgrade to GEM 2.1, we recommend you upgrade your Grafana Enterprise GEM plugin to the latest version. For more information on the most recent enhancements and fixes, see the plugin’s changelog page.

Bug fixes

v2.1.1

Fix CVE-2023-24538 by upgrading to Go 1.19.8

v2.1.0

Fixed a bug where ruler limits (e.g., maximum number of rules in a rule group or maximum number of rule groups per tenant) were not enforced when creating a rule group via GEM’s ruler API. These limits are now enforced.
GEM now returns a generic error message to clients (e.g., the Grafana Agent) upon internal errors during authentication. The detailed error message is now written to the GEM logs rather than being returned to the client.