Manage a tenant on Grafana Labs

Setting per-tenant resource usage limits

Fri, 03 Apr 2026 19:43:06 +0000

Overview

Per-tenant resource limits can be used to ensure a single tenant cannot monopolize the resources of or threaten the stability of a Grafana Enterprise Metrics cluster. There are limits available to tune usage of all parts of the metric read and write paths. For a complete list of all available limits, refer to the Grafana Mimir limits configuration.

Using the admin API of Grafana Enterprise Metrics, you can easily set a subset of these per-tenant limits.

The following limits are supported via the admin API:

ingestion_rate
ingestion_burst_size
max_series_per_query
max_global_series_per_user
max_global_series_per_metric
max_global_exemplars_per_user
ruler_max_rules_per_rule_group
ruler_max_rule_groups_per_tenant
max_fetched_chunks_per_query
max_fetched_series_per_query
max_fetched_chunk_bytes_per_query
compactor_blocks_retention_period

Setting limits via the admin API

Limits can be set using the admin API when creating a new tenant or when modifying an existing one.

Create a JSON payload, test1.json, for creating a new tenant that includes a subset of per-tenant limits.

{
  "name": "test1",
  "display_name": "Test Instance 1",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "max_series_per_query": 42000,
    "max_global_series_per_user": 42000,
    "compactor_blocks_retention_period": "24h"
  }
}

Use curl to POST the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants \
    -X POST --data @test1.json

Read the newly created tenant.

curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/test1 | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev",
  "limits": {
    "ingestion_rate": 350000,
    "ingestion_burst_size": 350000,
    "max_series_per_query": 42000,
    "max_global_series_per_user": 42000,
    "max_global_series_per_metric": 300000,
    "max_global_exemplars_per_user": 0,
    "ruler_max_rules_per_rule_group": 0,
    "ruler_max_rule_groups_per_tenant": 0,
    "max_fetched_chunks_per_query": 0,
    "max_fetched_series_per_query": 0,
    "max_fetched_chunk_bytes_per_query": 0,
    "compactor_blocks_retention_period": "24h"
  }
}

Note that all available limit settings are populated, even ones that we did not include in our POST request. Any limit settings not included when a new tenant is created with some limit settings are populated from the global default limits. This also applies when limits are added to a tenant that previously did not have them: any limit settings that are not included will be populated from the global default limits.

Removing limits via the admin API

Limits can be removed from a tenant using the admin API by modifying the tenant.

Create a JSON payload, test1-update.json, that doesn’t include the limits field.

{
  "display_name": "Test Instance 1",
  "status": "active",
  "cluster": "enterprise-metrics-dev"
}

Use curl to PUT the data to the admin API.

$ curl -u :$API_TOKEN localhost:8080/admin/api/v3/tenants/test1 \
    -X PUT --data @test1-update.json | jq
{
  "name": "test1",
  "display_name": "Test Instance 1",
  "created_at": "2020-07-13T17:37:59.341728283Z",
  "status": "active",
  "cluster": "enterprise-metrics-dev"
}

The modified tenant object is included in the update response.

Limit sources

Limits applied to a tenant are only taken from a single location at a time:

Per-tenant limits set by the admin API, backfilled with global defaults.
Per-tenant limits set by a runtime configuration file, backfilled with global defaults.
Global default limits set by a configuration file.

These locations are checked for limits to apply to operations, in that order. Once a location for limits to apply to an operation is found, no other locations are checked. The effect of this is that if you set limits for a tenant using the admin API, any limit settings for that tenant in a runtime configuration file will be ignored.

Implementation

Enforcing limits based on values set with the admin API requires periodically making requests to object storage (S3, GCS, etc.) to get the most up to date limits for each tenant. This means that based on the period you have configured to refresh tenant-based limits (1 minute by default), N GET requests and 1 LIST request will be made to object storage, where N is the number of tenants in your Grafana Enterprise Metrics cluster.

Using label-based access control (LBAC)

Fri, 03 Apr 2026 19:43:06 +0000

Overview

Label-based access control can be used to create access policies that will only allow for data to be queried that meets specific label requirements. The feature allows multiple sets of Prometheus label selectors to be associated with a policy and queries will only return data from series that match at least one of the provided selectors. This correlates to disjunctive normal form which allows any required policy to be expressed.

Setting up a label policy

Label policies are set when creating an access policy on a per tenant basis. This means each tenant associated with an access policy can have a unique label policy.

Examples

Exclude a label

One common use case for creating an LBAC policy is to exclude metrics with a specific label. For instance, a label policy that excludes all series with the label secret=true would be created by just adding a select with secret!="true" when creating an access policy. This can be seen in the image below:

Exclude a metric

Expanding upon the previous example, lets say we wanted to create an access policy that only excludes metrics with the label secret=true on the metric named sensitive_requests_total. Since the name of a metric is actually just a label with the key __name__, we can leverage the existing LBAC label selector syntax to enforce this:

You may notice above that two different selectors where added to enforce the policy. Specifically:

{secret!="true", __name__="sensitive_requests_total"}

and

{__name__!="sensitive_requests_total"}

The first selector will match when returning a series from the metrics sensitive_requests_total and will ensure all of the returned series do not have the secret: true label. However, when requesting a metric besides sensitive_requests_total, the second label selector will match and return any data even if it has the secret: true label.

Cardinality analysis

Fri, 03 Apr 2026 19:43:06 +0000

Overview

Grafana Enterprise Metrics provides the ability to understand the cardinality of your metrics and labels using Cardinality analysis dashboards that are shipped with the Grafana Enterprise Metrics plugin or via an API.

The APIs and dashboards help you understand the active time series in GEM. An active time series is one that has not yet been written to long-term storage.

Configuration

The API endpoints are disabled by default. Use one of the following approaches to enable or disable the endpoints for all tenants:

Add the CLI flag -querier.cardinality-analysis-enabled=true.
Set cardinality_analysis_enabled to true in the limits section of the global configuration file as shown below:

limits:
  cardinality_analysis_enabled: true

To selectively disable the endpoints for some tenants (if it’s been enabled for all tenants), or enable the endpoints for some tenants (when it is globally disabled), use the Runtime Configuration file.

Limitations

The cardinality analysis dashboards only work for single-tenant data sources. Similarly, the cardinality analysis APIs will only return cardinality information for a single tenant at a time. You cannot get a global view of the cardinality of multiple tenants simultaneously. This means that any call to the API where you provide multiple tenants in the username field will fail. For example, team-a|team-b will fail, but team-a or team-b will succeed.
The cardinality analysis dashboards do not work for data sources that use label-based access controls. Similarly, calls to the cardinality analysis APIs that use a token for an access policy with label selectors also fail.
The cardinality analysis APIs and dashboards will only work if you are running GEM using block storage. They are incompatible with chunks storage.

Operational considerations

We do not expect this new and experimental API to negatively affect the performance of ingesters in a GEM cluster. To be sure, monitor the cluster after enabling this feature.

To monitor the performance of the cardinality endpoints, use the exposed GEM API endpoints metrics.

The following example query returns the queries-per-second to the cardinality analysis endpoints:

sum by (route) (
    rate(cortex_querier_request_duration_seconds_sum{
        route=~"prometheus_api_v1_cardinality_label_values|prometheus_api_v1_cardinality_label_names"
    }[1m])
)

To monitor the performance of the whole cluster after enabling cardinality analysis, use the self-monitoring dashboards that are included in the GEM plugin.

Dashboards

The GEM plugin provides several useful dashboards that visualize and let you explore the data from this API.

Adding the cardinality analysis dashboards to Grafana Enterprise

The cardinality analysis dashboards are automatically installed if you install the Grafana Enterprise Metrics plugin. However, in the event that you do not see the dashboards or someone accidentally deletes them, add them back:

Go to Configuration > Plugins > Grafana Enterprise Metrics > Dashboards
Install the dashboards : Cardinality management - overview, Cardinality management - metrics and Cardinality management - labels.

Cardinality management - overview dashboard

This dashboard shows the cardinality for the selected data source.

Cardinality management - metrics dashboard

This dashboard helps you understand the cardinality of an individual metric. At the top of the dashboard, you can select which metric you want to explore.

Cardinality management - labels dashboard

This dashboard shows a cardinality report for the selected label. For a given label name, it shows you which label values are attached to the most series. It also shows you the highest cardinality metrics for a given label<>value pair.

HTTP API

You can use two API endpoints to understand a tenant’s metrics and label cardinality: label_names (/api/v1/cardinality/label_names) and label_values (/api/v1/cardinality/label_values). The cardinality analysis dashboards display information returned from these endpoints.

Because these endpoints generate their cardinality report using only values from currently opened TSDBs (time series databases) in the ingesters, two subsequent calls can return completely different results if an ingester cut or truncated an old block and opened a new one between calls.

Both API endpoints require authentication. Specifically, the user must provide a token which gives them metrics: read access for that tenant.

Label names cardinality endpoint

GET,POST <prometheus-http-prefix>/api/v1/cardinality/label_names

# Legacy
GET,POST <legacy-http-prefix>/api/v1/cardinality/label_names

Returns realtime label names cardinality across all ingesters, for the authenticated tenant, in JSON format. It counts distinct label values per label name.

The items in the field cardinality are sorted by label_values_count in descending order and by label_name in ascending order.

The count of items returned is limited by limit request parameter.

Request parameters

selector - optional - specifies PromQL selector that will be used to filter series that must be analyzed.
limit - optional - specifies max count of items in field cardinality in response (default=20, min=0, max=500)

Example:

To understand which labels attached to the metric flower_events_created have the most values, use the following command:

$ curl -u "<tenant-id>:$API_TOKEN" "<host and port>/prometheus/api/v1/cardinality/label_names?limit=2&selector=\{__name__='flower_events_created'\}" | jq

{
  "label_values_count_total": 206,
  "label_names_count": 12,
  "cardinality": [
    {
      "label_name": "worker",
      "label_values_count": 162
    },
    {
      "label_name": "task",
      "label_values_count": 29
    }
  ]
}

From this we see that the metric flower_events_created has 12 different label names attached to it. Across those 12 label names, there are 206 total values. The label “worker” has 162 values, and the label “task” has 29 values. Not shown are the other label names, since the sample command set limit=2.

If the flower_events_created selector were omitted, the API call

$ curl -u "<tenant-id>:$API_TOKEN" "<host and port>/prometheus/api/v1/cardinality/label_names?limit=2" | jq

would return the label names with the highest count of values across the entire tenant.

Response schema

{
  "label_values_count_total": <number>,
  "label_names_count": <number>,
  "cardinality": [
    {
      "label_name": <string>,
      "label_values_count": <number>
    }
  ]
}

Label values cardinality endpoint

GET,POST <prometheus-http-prefix>/api/v1/cardinality/label_values

# Legacy
GET,POST <legacy-http-prefix>/api/v1/cardinality/label_values

Returns realtime label values cardinality associated with request parameter label_names[] across all ingesters, for the authenticated tenant, in JSON format. It returns the series count per label value for each label in the request parameter label_names[].

The items in the field labels are sorted by series_count in descending order and by label_name in ascending order. The items in the field cardinality are sorted by series_count in descending order and by label_value in ascending order.

The count of cardinality items is limited by request parameter limit.

Request parameters

label_names[] - required - specifies labels for which cardinality must be provided.
selector - optional - specifies PromQL selector that will be used to filter series that must be analyzed.
limit - optional - specifies max count of items in field cardinality in response (default=20, min=0, max=500).

Example 1 (label values cardinality):

In case we want to understand which label values have the highest number of flower_events_created series associated with them, we can execute:

$ curl -u "<tenant-id>:$API_TOKEN" "<host and port>/prometheus/api/v1/cardinality/label_values?label_names[]=worker&label_names[]=agent&limit=2&selector=\{__name__='flower_events_created'\}" | jq

{
  "series_count_total": 5472781,
  "labels": [
    {
      "label_name": "worker",
      "label_values_count": 162,
      "series_count": 1307,
      "cardinality": [
        {
          "label_value": "aws-worker",
          "series_count": 67
        },
        {
          "label_value": "gcp-worker",
          "series_count": 66
        }
      ]
    },
    {
      "label_name": "agent",
      "label_values_count": 2,
      "series_count": 11,
      "cardinality": [
        {
          "label_value": "grafana-agent",
          "series_count": 10
        },
        {
          "label_value": "jaeger-agent",
          "series_count": 1
        }
      ]
    }
  ]
}

From this, we see that there are 5,472,781 series with the metric name flower_events_created. Of those 5,472,781 series, there are 67 series with worker=aws-worker and 66 series with worker=gcp-worker. From the series_count, there are 1307 series with the label worker (across all 162 values of worker).

Similarly, of the 5,472,781 total series, there are 10 series with agent=grafana-agent and 1 series with agent=jaeger-agent. From the series_count there are 11 total series with the label agent (across all 2 values of agent).

Example 2 (metric names cardinality):

In case we want to understand which metrics have the highest cardinality (i.e. have the most time series) you can look at the cardinality of the __name__ label.

$ curl -u "<tenant-id>:$API_TOKEN" "<host and port>/prometheus/api/v1/cardinality/label_values?label_names[]=__name__&limit=2" | jq

{
  "series_count_total": 1307,
  "labels": [
    {
      "label_name": "__name__",
      "label_values_count": 162,
      "series_count": 1307,
      "cardinality": [
        {
          "label_value": "flower_events_created",
          "series_count": 67
        },
        {
          "label_value": "flower_events_consumed",
          "series_count": 66
        }
      ]
    }
  ]
}

In this example, there are 1307 total active time series for the tenant named tenant-id. As there are 162 values for the label __name__, we know this means there are 162 metrics for this tenant. The label_value in the cardinality part of the payload are the names of the highest cardinality metrics. In this example, we see that metric flower_events_created has 67 series associated with it and metric flower_events_consumed has 66 series associated with it.

Response schema

{
  "series_count_total": <number>,
  "labels": [
    {
      "label_name": <string>,
      "label_values_count": <number>,
      "series_count": <number>,
      "cardinality": [
        {
          "label_value": <string>,
          "series_count": <number>
        }
      ]
    }
  ]
}

series_count_total - total number of series across opened TSDBs in all ingesters
labels[].label_name - label name requested via the request parameter label_names[]
labels[].label_values_count - total number of label values for the label name (note that dependent on the limit request parameter it is possible that not all label values are present in cardinality)
labels[].series_count - total number of series having labels[].label_name
labels[].cardinality[].label_value - label value associated with labels[].label_name
labels[].cardinality[].series_count - total number of series having label_value for label_name

Using tenant federation

Fri, 03 Apr 2026 19:43:06 +0000

Overview

GEM supports creating access policies that can span multiple tenants. Doing so enables viewers in Grafana Enterprise to view data coming from more than one tenant simultaneously.

This page will cover the ability to query data from multiple tenants at once, as well as the ability to create alerting and recording rules that use data from multiple tenants at once.

Prerequisites

A configured Grafana Enterprise Metrics cluster. To create a GEM cluster, refer to Set up GEM.
This guide will assume there are two tenants: team-engineering and team-finance. To create a tenant, refer to Set up a GEM tenant.

Cross-tenant queries

Set up an access policy with tenant federation and a token

To allow queries to span both GEM tenants, which are for demonstration purposes named team-engineering and team-finance, create a new access policy called leadership. The necessary steps are:

Create a new access policy leadership.
Enable the metrics:read scope.
Add the tenants team-engineering and team-finance. Alternatively, you can add the special tenant name * to create an access policy that has access to all tenants in the cluster.
Create a new token for the access-policy and store the token in your clipboard:

Set up a Grafana data source using the access policy

Create a new Prometheus data source from the Grafana configuration menu.
Enter the URL of your GEM cluster, for example http://enterprise-metrics/prometheus.
From the Auth section, enable Basic auth.
In the User field, enter: team-engineering|team-finance where all the names of the tenants that you want to query across are separated by the | pipe character.
In the Password field, paste the token created in the token creation process.

Queries that are performed using this data source in either Explore or inside of dashboards are performed across all of the tenants that you specified in the User field, and are processed as if all of the data were in a single tenant.

To submit a query across all tenants that your access policy has access to, you can either:

Explicitly set the name of all the tenants separated by a pipe character “|” in the username. For example, to query across tenant1, tenant2, and tenant3 you would enter tenant1|tenant2|tenant3.
Set the username to a wildcard character *. This will query all tenants that the access policy grants you access to, without requiring you to explicitly specify their names.

When using an access policy that has a wildcard (*) as the username, you can query all tenants for that cluster by also specifying * as the username in your data source URL.

Conversely, if you use a wildcard username in your data source configuration with an access policy with specific tenants, that data source has access to only those tenants.

Cross-tenant alerting and recording rules

Cross-tenant alerting and recording rules are an experimental feature.

GEM supports alerting and recording rules that operate on data from multiple tenants.

To create a federated alerting or recording rule, use the source_tenants field when defining your rule group. This field defines which tenants the rule will read data from. Any rule in a group with a non-empty source_tenants field is considered a federated rule.

Using the same example tenants we defined above, we’ll create a federated rule group that evaluates data coming from tenants team-engineering and team-finance, and stores the resulting series in team-engineering.

First, create your rule and rule group in a file called payload.yaml.

name: example
interval: 5m
source_tenants: [team-engineering, team-finance]
rules:
  - record: job:http_inprogress_requests:sum
    expr: sum by (job) (http_inprogress_requests)

This example recording rule is calculating a sum of http_inprogress_requests by job every 5 minutes. Because the source tenants are team-engineering and team-finance, GEM will use all series with metric name http_inprogress_requests in both of these tenants when calculating the sum.

To create the cross-tenant rule group, use the curl command shown below, which uses the Set rule group API endpoint and passes in the payload.yaml we defined above:

curl -u team-engineering:$API_TOKEN -X POST http://enterprise-metrics/api/v1/rules/fed-rule-group --data-binary "@/payload.yaml"

The username used in the curl call is what dictates where the output of the rule will be stored. In our example above, the output of the rule will be stored in team-engineering because we have -u team-engineering.

The access policy to which the API token belongs must have metrics:read and rules:write scopes and apply to all of the tenants in source_tenants (team-engineering and team-finance in our example) and the owning tenant (team-engineering in our example). Otherwise, an HTTP unauthorized status (401) will be returned and the rule group will not be created.

When the access policy used to create the federated rule group is deleted or updated to remove the metrics:read scope or to remove any of the source_tenants, that rule group will stop evaluating at the next rules synchronization. The synchronization interval is set by the -ruler.poll-interval configuration parameter. By default, this is 1 minute.

You can find more details about cross-tenant rules in the Grafana Mimir API documentation.