Set up a tenant in your Grafana Enterprise Traces cluster

Tenants provide a mechanism for trace isolation. Access policies may be set on a per-tenant basis. Authorization of requests is based on specified access policies.

These instructions assume that you have the Grafana Enterprise Traces (GET) plugin installed. Use this plugin to create tenants, access policies, and tokens for your GET cluster. The steps include:

Create a GET tenant.
Create an access policy.
Configure remote-write to your tenant.
Set up your tenant as a Grafana data source.
Visualize your data.

Create a GET tenant

Once a cluster is running, new tenants may be created. To create a GET tenant:

Within Grafana Enterprise, navigate to Apps > Grafana Enterprise Traces > Tenants.
Click on Create tenant.
Choose a name for this tenant. For demonstration purposes, use the name dev-tenant.
Select the cluster.
Click Create tenant.

Create an access policy

Access policies are used to authorize actions and operations by specified tenants. Access policies have a realm, which defines the set of tenants they apply to, and a scope, which defines the set of actions upon which they confer permissions to use.

Grafana Enterprise requires a data source access policy and token to access the traces in the tenant named dev-tenant.

Navigate to Apps > Grafana Enterprise Traces > Access policies.
Click Create access policy.
Choose a name for the policy. For demonstration purposes, use dev-read-write-policy.
Under Scopes, select the toggle next to the question Are you planning on creating a data source that uses this access policy?. This enables the traces:read policy.
Enable the scope traces:write to create an access policy that grants access for both reading and writing trace data (as the previous step toggles the traces:read policy automatically).
Select the tenant dev-tenant.
Click on Create.
From the newly created access policy, click Add token.
Name the token dev-token and click on Create.
In the next window, copy the token by selecting Copy to clipboard.

At this point, you can add a data source to your Grafana Enterprise instance by using the Create a data source option on this dialog or by manually adding one using the procedure below.

Select Create a data source. The data source fields are automatically configured based upon your instance.
Verify the created data source by opening the Configuration option in the Grafana menu and then selecting Data sources.

Set up remote-write to your tenant

To enable writes to your cluster, add a remote-write configuration snippet to the configuration file of an existing Grafana Agent. If you do not have an existing traces collector, refer to Set up with Grafana Agent. For Kubernetes, refer to the Grafana Agent Traces Kubernetes quick start guide.

An example agent configuration would be:

traces:
  configs:
  - name: default
    receivers:
      jaeger:
        protocols:
          grpc:           # listens on the default jaeger grpc port: 14250
          thrift_binary:  # 6832
          thrift_compact: # 6831
          thrift_http:    # 14268
    remote_write:
      - endpoint: <get-gateway-host>:<http_listen_port>
        insecure: true  # only add this if TLS is not required
        basic_auth:
          username: dev-tenant
          password: ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ==
    batch:
      timeout: 5s

Create a Grafana data source

To allow Grafana to read traces from GET, you must create a Tempo data source with the proper credentials.

Navigate to Configuration ≫ Data Sources.
Click on Add new data source.
Select Tempo.
Set the URL to http://<get-gateway-host>:<http_listen_port>/<http_api_prefix>, filling in the path to your gateway and the configured HTTP API prefix.
Enable Basic Auth. Use User dev-tenant and the token from your clipboard as the Password.
Click Save & Test.

You should see a message that says Data source is working.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page.

Test your configuration using the TNS application

You can use The New Stack (TNS) application to test GET data.

Navigate to https://github.com/grafana/tns/tree/main/production/k8s-yamls to get the Kubernetes manifests for the TNS application.

Clone the repository using commands similar to the ones below:

  mkdir ~/tmp
  git clone git+ssh://github.com/grafana/tns
  cp tns/production/k8s-yamls/* <targetDir>

Change to the new directory: cd <targetDir>
In each of the -dep.yaml manifests, alter the JAEGER_AGENT_HOST to the Grafana Agent location. For example, based on the above Grafana Agent install:
yaml
```
 	Env:
 	- name: JAEGER_AGENT_HOST
   value: grafana-agent-traces.default.svc.cluster.local
```

Deploy the TNS application. It will deploy into the default namespace.

    kubectl apply -f app-svc.yaml,db-svc.yaml,loadgen-svc.yaml,app-dep.yaml,db-dep.yaml,loadgen-dep.yaml

Once the application is running, look at the logs for one of the pods (such as the App pod) and find a relevant trace ID. For example:

 kubectl logs $(kubectl get pod -l name=app -o jsonpath="{.items[0].metadata.name}")
 level=debug traceID=50075ac8b434e8f7 msg="GET / (200) 1.950625ms"
 level=info msg="HTTP client success" status=200 url=http://db duration=1.297806ms traceID=2c2fd669c388e76
 level=debug traceID=2c2fd669c388e76 msg="GET / (200) 1.70755ms"
 level=info msg="HTTP client success" status=200 url=http://db duration=1.853271ms traceID=79058bb9cc39acfb
 level=debug traceID=79058bb9cc39acfb msg="GET / (200) 2.300922ms"
 level=info msg="HTTP client success" status=200 url=http://db duration=1.381894ms traceID=7b0e0526f5958549
 level=debug traceID=7b0e0526f5958549 msg="GET / (200) 2.105263ms"

Go to Grafana Enterprise and select the Explore menu item.
Select the GET data source from the list of data sources.
Copy the trace ID into the Trace ID edit field.
Select Run query.
The trace will be displayed in the traces Explore panel.

Feedback