Overview
GEM supports creating access policies that can span multiple tenants. Doing so enables viewers in Grafana Enterprise to view data coming from more than one tenant simultaneously.
This page will cover the ability to query data from multiple tenants at once, as well as the ability to create alerting and recording rules that use data from multiple tenants at once.
Prerequisites
A configured Grafana Enterprise Metrics cluster. To create a GEM cluster, refer to Set up GEM.
This guide will assume there are two tenants:
team-engineering
andteam-finance
. To create a tenant, refer to Set up a GEM tenant.
Cross-tenant queries
Set up an access policy with tenant federation and a token
To allow queries to span both GEM tenants, which are for demonstration purposes named team-engineering
and team-finance
,
create a new access policy called leadership
. The necessary steps are:
Create a new access policy
leadership
.Enable the
metrics:read
scope.Add the tenants
team-engineering
andteam-finance
. Alternatively, you can add the special tenant name*
to create an access policy that has access to all tenants in the cluster.Create a new token for the access-policy and store the token in your clipboard:
Set up a Grafana data source using the access policy
Create a new Prometheus data source from the Grafana configuration menu.
Enter the URL of your GEM cluster, for example
http://enterprise-metrics/prometheus
.From the Auth section, enable Basic auth.
In the User field, enter:
team-engineering|team-finance
where all the names of the tenants that you want to query across are separated by the|
pipe character.In the Password field, paste the token created in the token creation process.
Queries that are performed using this data source in either Explore or inside of dashboards are performed across all of the tenants that you specified in the User field, and are processed as if all of the data were in a single tenant.
To submit a query across all tenants that your access policy has access to, you can either:
- Explicitly set the name of all the tenants separated by a pipe character “|” in the username. For example, to query across
tenant1
,tenant2
, andtenant3
you would entertenant1|tenant2|tenant3
. - Set the username to a wildcard character
*
. This will query all tenants that the access policy grants you access to, without requiring you to explicitly specify their names.
When using an access policy that has a wildcard (*
) as the username, you can query all tenants for that cluster by also specifying *
as the username in your data source URL.
Conversely, if you use a wildcard username in your data source configuration with an access policy with specific tenants, that data source has access to only those tenants.
Cross-tenant alerting and recording rules
Cross-tenant alerting and recording rules are an experimental feature.
GEM supports alerting and recording rules that operate on data from multiple tenants.
To create a federated alerting or recording rule, use the source_tenants
field when defining your rule group. This field defines which tenants the rule will read data from. Any rule in a group with a non-empty source_tenants
field is considered a federated rule.
Using the same example tenants we defined above, we’ll create a federated rule group that evaluates data coming from tenants team-engineering
and team-finance
, and stores the resulting series in team-engineering
.
First, enable query tenant federation and rules tenant federation. The configuration parameters for each are as follows:
CLI argument | YAML parameter |
---|---|
-ruler.tenant-federation.enabled=true | ruler.tenant_federation.enabled: true |
-tenant-federation.enabled=true | tenant_federation.enabled: true |
If you are using the mimir-distributed
Helm chart to deploy GEM, then you can enable these parameters by including the following configuration in your values file:
mimir:
structuredConfig:
tenant_federation:
enabled: true
ruler:
tenant_federation:
enabled: true
Next, create your rule and rule group in a file named groups.yaml
.
namespace: fed-rule-group
groups:
- name: example
interval: 5m
source_tenants: [team-engineering, team-finance]
rules:
- record: job:http_inprogress_requests:sum
expr: sum by (job) (http_inprogress_requests)
This example recording rule is calculating a sum of http_inprogress_requests
by job every 5 minutes. Because the source tenants are team-engineering
and team-finance
, GEM will use all series with metric name http_inprogress_requests
in both of these tenants when calculating the sum.
To create the cross-tenant rule group, use the following mimirtool command:
mimirtool rules load --address http://enterprise-metrics/ --user team-engineering --key $API_TOKEN groups.yaml
The username used in the mimirtool command call is what dictates where the output of the rule will be stored. In our
example above, the output of the rule will be stored in team-engineering
because we have --user team-engineering
.
The access policy to which the API token belongs must have metrics:read
and rules:write
scopes and apply to all the tenants in source_tenants
(team-engineering
and team-finance
in our example) and the owning tenant (team-engineering
in our example). Otherwise, an HTTP unauthorized status (401
) is returned and the rule group is not created.
Note: Creating federated rules is not supported with OIDC-based tokens that reference more than one GEM access policy. To create federated rules, either use a GEM token or an OIDC-based token with only a single GEM access policy that has
metrics:read
andrules:write
scopes and applies to the owning tenant and the source tenants.
When the access policy used to create the federated rule group is deleted or updated to remove the metrics:read
scope or to remove any of the source_tenants
, that rule group will stop evaluating at the next rules synchronization. The synchronization interval is set by the -ruler.poll-interval
configuration parameter. By default, this is 1 minute.
You can find more details about cross-tenant rules in the Grafana Mimir API documentation.