GEM supports creating access policies that can span multiple tenants. Doing so enables viewers in Grafana Enterprise to view data coming from more than one tenant simultaneously.
This page will cover the ability to query data from multiple tenants at once, as well as the ability to create alerting and recording rules that use data from multiple tenants at once.
A configured Grafana Enterprise Metrics cluster. To create a GEM cluster, refer to Set up GEM.
This guide will assume there are two tenants:
team-finance. To create a tenant, refer to Set up a GEM tenant.
Set up an access policy with tenant federation and a token
To allow queries to span both GEM tenants, which are for demonstration purposes named
create a new access policy called
leadership. The necessary steps are:
Create a new access policy
Add the tenants
team-finance. Alternatively, you can add the special tenant name
*to create an access policy that has access to all tenants in the cluster.
Create a new token for the access-policy and store the token in your clipboard:
Set up a Grafana data source using the access policy
Create a new Prometheus data source from the Grafana configuration menu.
Enter the URL of your GEM cluster, for example
From the Auth section, enable Basic auth.
In the User field, enter:
team-engineering|team-financewhere all the names of the tenants that you want to query across are separated by the
In the Password field, paste the token created in the token creation process.
Queries that are performed using this data source in either Explore or inside of dashboards are performed across all of the tenants that you specified in the User field, and are processed as if all of the data were in a single tenant.
To submit a query across all tenants that your access policy has access to, you can either:
- Explicitly set the name of all the tenants separated by a pipe character “|” in the username. For example, to query across
tenant3you would enter
- Set the username to a wildcard character
*. This will query all tenants that the access policy grants you access to, without requiring you to explicitly specify their names.
When using an access policy that has a wildcard (
*) as the username, you can query all tenants for that cluster by also specifying
* as the username in your data source URL.
Conversely, if you use a wildcard username in your data source configuration with an access policy with specific tenants, that data source has access to only those tenants.
Cross-tenant alerting and recording rules
Cross-tenant alerting and recording rules are an experimental feature.
GEM supports alerting and recording rules that operate on data from multiple tenants.
To create a federated alerting or recording rule, use the
source_tenants field when defining your rule group. This field defines which tenants the rule will read data from. Any rule in a group with a non-empty
source_tenants field is considered a federated rule.
Using the same example tenants we defined above, we’ll create a federated rule group that evaluates data coming from tenants
team-finance, and stores the resulting series in
First, enable query tenant federation and rules tenant federation. The configuration parameters for each are as follows:
If you are using the
mimir-distributed Helm chart to deploy GEM, then you can enable these parameters by including the following configuration in your values file:
Next, create your rule and rule group in a file named
- name: example
source_tenants: [team-engineering, team-finance]
- record: job:http_inprogress_requests:sum
expr: sum by (job) (http_inprogress_requests)
This example recording rule is calculating a sum of
http_inprogress_requests by job every 5 minutes. Because the source tenants are
team-finance, GEM will use all series with metric name
http_inprogress_requests in both of these tenants when calculating the sum.
To create the cross-tenant rule group, use the following mimirtool command:
mimirtool rules load --address http://enterprise-metrics/ --user team-engineering --key $API_TOKEN groups.yaml
The username used in the mimirtool command call is what dictates where the output of the rule will be stored. In our
example above, the output of the rule will be stored in
team-engineering because we have
The access policy to which the API token belongs must have
rules:write scopes and apply to all the tenants in
team-finance in our example) and the owning tenant (
team-engineering in our example). Otherwise, an HTTP unauthorized status (
401) is returned and the rule group is not created.
Note: Creating federated rules is not supported with OIDC-based tokens that reference more than one GEM access policy. To create federated rules, either use a GEM token or an OIDC-based token with only a single GEM access policy that has
rules:writescopes and applies to the owning tenant and the source tenants.
When the access policy used to create the federated rule group is deleted or updated to remove the
metrics:read scope or to remove any of the
source_tenants, that rule group will stop evaluating at the next rules synchronization. The synchronization interval is set by the
-ruler.poll-interval configuration parameter. By default, this is 1 minute.
You can find more details about cross-tenant rules in the Grafana Mimir API documentation.