Grafana Cloud Connect to data sources Multi-stack data sources

Query data across multiple stacks

Overview

Grafana Cloud supports the creation of access policies that span multiple stacks, enabling simultaneous data querying across different stacks. This document walks you through the creation of an access policy and data sources needed to perform these multi-stack queries.

Set up a multi-stack access policy and a generate a token

For demonstration purposes, consider two stacks named teamengineering and teamfinance located in the same region. Follow the steps below to create an access policy that enables reading metrics and logs from these stacks:

1. Under the Access Policies section in the Cloud Portal, create an access policy called leadership.
2. Choose teamengineering and teamfinance as realms. Alternatively, you can opt for the “All stacks” realm to create a policy that has access to all stacks.
3. Select the metrics:read and logs:read scopes.
4. After creating the policy, generate a new token and keep it handy for a later step.

Set up a multi-stack Prometheus data source in Grafana

1. In the Cloud Portal, navigate to the Prometheus details of both teamengineering and teamfinance stacks and make note of their instance URLs and IDs.
2. In Grafana, create a new Prometheus data source.
3. Enter the URL noted down in step 1 (e.g. https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom).
4. Select Basic Authentication as the authentication method.
5. In the User field, enter the IDs obtained in step 1, separated by the | character (e.g. 1|2).
6. Paste the previously generated token into the Password field.

Set up a multi-stack Loki data source in Grafana

1. In the Cloud Portal, navigate to the Loki details of both teamengineering and teamfinance stacks and make note of their instance URLs and IDs.
2. In Grafana, create a new Loki data source.
3. Enter the URL noted down in step 1 (e.g. https://logs-prod-006.grafana.net).
4. Select Basic Authentication as the authentication method.
5. In the User field, enter the IDs obtained in step 1, separated by the | character (e.g. 1|2).
6. Paste the previously generated token into the Password field.

Query your multi-stack data source

You can query your multi-stack data source as you would any other Prometheus or Loki data source via the Explore page or by creating a dashboard. The queries you make to your multi-stack data source will encompass all specified stacks in the User field, processing data as if it were in a unified stack.

Query responses from your multi-stack data source will include a __tenant_id__ label, indicating the originating stack’s Prometheus or Loki instance.

Query data from all stacks authorized by an access policy

For querying across all stacks within a region, begin by selecting the All stacks realm when creating the access policy, Next, use the * wildcard in the User field when setting up the data source. The data source you create will query all stacks in the region specified in your URL.

Conversely, you can use a wildcard * in the User field of your data source in combination with an access policy with access restricted to specific stacks. In this case, the multi-stack data source will have access to all stacks in that access policy’s realm.

Limitations

• During public preview, cross-stack querying is constrained to a maximum of 10 stacks. If you need to query over greater number of stacks, we encourage you to get in touch with the Grafana Support team for assistance.