Cluster query federation
The optional federation-frontend
service provides the ability to aggregate
trace data from multiple Grafana Enterprise Traces clusters in a single trace
lookup.
The federation-frontend
target itself does not require any other components
of GET, therefore you can run it on its own. A common use case for query
federation is aggregating the data from two or more GET clusters that are
running in different regions or data centers.
Configuration
To start the federation frontend, configure the target
option to be
federation-frontend
.
A minimal configuration of the federation-frontend
target has authentication disabled
, because it forwards the HTTP authentication header
(Authorization: Basic ...
) that is supplied by its clients to the target
clusters.
You need to configure a list of target clusters within the
federation.proxy_targets
block; currently, there are no equivalent CLI flags
available. Each entry requires a url
that points to a Tempo compatible API.
For Grafana Enterprise Traces clusters, use the URL
http://<get-host>:<get-port>/<http_api_prefix>
.
Optionally, you can configure each proxy_target
to have Basic auth
credentials, which override the user-supplied ones.
When you configure Basic auth via the proxy_target
configuration, these
credentials take precedence over the ones supplied by the client. Without
enabling authentication on the federation frontend, any client that can reach
the federation frontend can perform queries on the target clusters by using
those credentials.
In the following example, two clusters in two different regions are queried via the federation frontend:
auth_enabled: false # The federation frontend does not do any authentication itself
target: federation-frontend # Run the federation frontend only
federation:
proxy_targets:
- url: http://gem-us-west/tempo
- url: http://gem-us-east/tempo
Aggregate traces from a local GET cluster and Grafana Cloud Traces instance
The federation frontend allows you to get an aggregated view of traces stored in a local GET cluster and a hosted Grafana Cloud Traces instance. With the following configuration, you can query both of the clusters as though they were one:
federation:
proxy_targets:
- name: own-data-center
url: http://gem/tempo
- name: grafana-cloud
url: https://tempo-us-central1.grafana.net/tempo
basic_auth:
username: <instance-id>
password: <token>
Warning: This gives any client that can reach the federation frontend access to your traces data in Grafana Cloud Traces without further authentication.
By using the authentication credentials of the local GET cluster in your query request, you can execute a query against both clusters. To do so, set the access policy’s token as a variable for subsequent commands:
export API_TOKEN="the long token string you copied"
curl -s -u "<tenant-id>:$API_TOKEN" http://federation-frontend/tempo/api/traces/4d65822107fcfd52 | jq
The federation frontend will pass the credentials from your query command to cluster=“own-data-center”. Because the basic_auth block has been configured to include credentials for the “grafana-cloud” cluster, the credentials supplied in the query command will be overridden with those from the configuration file. The credentials in the configuration file will be used to authenticate with the “grafana-cloud” cluster.
Limitations of cluster query federation
This feature comes with several limitations:
- No result caching in the federation frontend
- No support for traces search
If your use-case is blocked by one of those limitations, please reach out through our support channels with a feature request.