Cross-tenant querying
GET supports creating access policies that can span multiple tenants. Viewers in Grafana Enterprise can see data coming from more than one tenant simultaneously.
This page covers the ability to query data from multiple tenants at once.
Prerequisites
A configured Grafana Enterprise Traces cluster. To create a GET cluster, refer to Set up GET.
This guide assumes there are two tenants:
team-engineeringandteam-finance. To create a tenant, refer to Set up a GET tenant.
Cross-tenant queries
This section describes cross-tenant queries.
Federation frontend
Tenant federation is handled by the federation-frontend service.
This service aggregates the data from multiple tenants in a single trace lookup or search query.
The federation-frontend runs with tenant federation enabled by default.
Queries with that contain the header X-Org-Id with multiple tenants separated by the | character,
are aggregated across all of specified tenants.
Configuration
To run the federation frontend, configure the target option to be federation-frontend.
Then, you only need to indicate a proxy target to which the federation frontend will forward the queries.
target: federation-frontend # Run the federation frontend only
federation:
proxy_targets:
- url: http://get-us-west/tempo
Set up an access policy with tenant federation and a token
To allow queries to span both GET tenants, create a new access policy called leadership.
For demonstration purposes, these tenants are named team-engineering and team-finance.
To create a raw access policy:
- Create a new access policy
leadership. - Enable the
traces:readscope. - Add the tenants
team-engineeringandteam-finance. Alternatively, you can add the special tenant name*to create an access policy that has access to all tenants in the cluster. - Create a new token for the access-policy and store the token in your clipboard:
There’s supposed to be a video here, but for some reason there isn’t. Either we entered the id wrong (oops!), or Vimeo is down. If it’s the latter, we’d expect they’ll be back up and running soon. In the meantime, check out our blog!
Set up a Grafana data source using the access policy
To set up a data source using the access policy:
- Create a new Tempo data source from the Grafana configuration menu.
- Enter the URL of your GET cluster, for example
http://enterprise-traces:3200. - From the Auth section, enable Basic auth.
- In the User field, enter:
team-engineering|team-financewhere all the names of the tenants that you want to query across are separated by the|pipe character. - In the Password field, paste the token created in the token creation process.
Queries that are performed using this data source in either Explore or inside of dashboards are performed across all the tenants that you specified in the User field. These queries are processed as if all the data were in a single tenant.
To submit a query across all tenants that your access policy has access to, you can either:
Explicitly set the name of all the tenants separated by a pipe character “|” in the username. For example, to query across
tenant1,tenant2, andtenant3you would entertenant1|tenant2|tenant3.Set the username to a wildcard character
*. This will query all tenants that the access policy grants you access to, without requiring you to explicitly specify their names.
When using an access policy that has a wildcard (*) as the username,
you can query all tenants for that cluster by also specifying * as the username in your data source URL.
Conversely, if you use a wildcard username in your data source configuration with an access policy with specific tenants, that data source has access to only those tenants.
