Set up a tenant in your Grafana Enterprise Traces cluster

Create a Grafana Enterprise Traces (GET) tenant:

1. Within Grafana Enterprise, navigate to Grafana Enterprise Traces > Tenants.

2. Click on Create new tenant.

3. Choose a name for this tenant. For demonstration purposes, use the name dev-tenant.

4. Select the cluster.

5. Click Create tenant.

Create an access policy

Create a data source access policy and token, which are used by Grafana Enterprise itself to access the traces in the tenant named dev-tenant:

1. Navigate to Grafana Enterprise Traces > Access policies.

2. Click Create new access policy.

3. Choose a name for the policy. For demonstration purposes, use dev-read-write-policy.

4. Enable the scope traces:read and traces:write, to create an access policy that grants access for both reading and writing trace data.

5. Select the tenant dev-tenant.

6. Click on Create.

7. From the newly created access policy, click + Add token.

8. Name the token dev-token, and click on Create.

9. In the next window, copy the token.

You can add the following example of a remote-write configuration to your Grafana Agent configuration files:

remote_write:
  - endpoint: <get-gateway-host>:<http_listen_port>
    insecure: true
    basic_auth:
      username: dev-tenant
      password: ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ==

Setup remote-write to your tenant

To enable writes to your cluster, add the above remote-write configuration snippet to the configuration file of an existing Grafana Agent. If you do not have an existing traces collector, refer to Getting started with Grafana Agent.

An example agent configuration would be:

server:
  http_listen_port: 12345

traces:
  configs:
  - name: default
    receivers:
      jaeger:
        protocols:
          grpc:           # listens on the default jaeger grpc port: 14250
          thrift_binary:  # 6832
          thrift_compact: # 6831
          thrift_http:    # 14268
    remote_write:
      - endpoint: <get-gateway-host>:<http_listen_port>
        insecure: true  # only add this if TLS is not required
        basic_auth:
          username: dev-tenant
          password: ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ==
    batch:
      timeout: 5s

Set up your tenant as a Grafana datasource

Following that we can use this access policy and token to create a new datasource in Grafana:

1. Navigate to Configuration ≫ Data Sources.

2. Click on Add new data source.

3. Select Tempo.

4. Set the URL to http://<get-gateway-host>:<http_listen_port>/<http_api_prefix>, filling in the path to your gateway and the configured HTTP API prefix.

5. Enable Basic Auth and use User dev-tenant and as Password the token from your clipboard.

6. Click Save & Test. You should see a message that says “Data source is working.” If you see an error that says “Data source is not working: failed to get trace with id: 0”, check your Grafana version. If your Grafana version is < 8.2.2, this is a bug in Grafana itself and the error can be ignored. The underlying data source will still work as expected. Upgrade your Grafana to 8.2.2 or later to get the fix for the error message.

Visualize your data

Once you have created a datasource you should now be able to visualize your traces in the Grafana Explore page.