Using tenant and cluster query federation
Grafana Enterprise Traces (GET) can perform cross-tenant and cross-cluster queries using the optional
This service aggregates the data from multiple GET clusters and from multiple tenants in a single trace lookup or search query.
It works by reverse proxying the read path of GET clusters and merging the results into a single response.
federation-frontend target itself requires a configured GET installation.
No other GET components are required.
You can run the
federation-frontend service on its own.
To read more about tenant federation, see the Cross-tenant querying page.
The diagram below shows the cluster federation architecture.
Set up the federation frontend
To start the federation frontend, configure the
target option to be
A minimal configuration of the
federation-frontend target has authentication disabled,
because it forwards the HTTP authentication header
Authorization: Basic ...) that is supplied by its clients to the target
You need to configure a list of target clusters within the
federation.proxy_targets block; currently, there are no equivalent CLI flags
available. Each entry requires a
url that points to a Tempo compatible API.
For Grafana Enterprise Traces clusters, use the URL
Optionally, you can configure each
proxy_target to have Basic auth
credentials, which override the user-supplied ones.
When you configure Basic auth via the
proxy_target configuration, these
credentials take precedence over the ones supplied by the client. Without
enabling authentication on the federation frontend, any client that can reach
the federation frontend can perform queries on the target clusters by using
In the following example, two clusters in two different regions are queried via the federation frontend:
auth_enabled: false # The federation frontend does not do any authentication itself target: federation-frontend # Run the federation frontend only federation: proxy_targets: - url: http://get-us-west/tempo - url: http://get-us-east/tempo
Aggregate traces from a local GET cluster and Grafana Cloud Traces instance
The federation frontend provides an aggregated view of traces stored in a local GET cluster and a hosted Grafana Cloud Traces instance. With the following configuration, you can query both of the clusters as though they were one:
federation: proxy_targets: - name: own-data-center url: http://get/tempo - name: grafana-cloud url: https://tempo-us-central1.grafana.net/tempo basic_auth: username: <instance-id> password: <token>
Warning: This gives any client that can reach the federation frontend access to your traces data in Grafana Cloud Traces without further authentication.
By using the authentication credentials of the local GET cluster in your query request, you can execute a query against both clusters. To do so, set the access policy’s token as a variable for subsequent commands:
export API_TOKEN="the long token string you copied" curl -s -u "<tenant-id>:$API_TOKEN" http://federation-frontend/tempo/api/traces/4d65822107fcfd52 | jq
The federation frontend passes the credentials from your query command to
basic_auth block has been configured to include credentials for the
grafana-cloud cluster, the credentials supplied in the query command are overridden with those from the configuration file.
The credentials in the configuration file will be used to authenticate with the
Limitations of the federation frontend
This feature has one known limitation:
- No result caching in the federation frontend
If your use-case is blocked by one of those limitations, please reach out through our support channels with a feature request.