Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Set up

Set up a test application for a Tempo cluster

Open source

Set up a test application for a Tempo cluster

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana. This procedure uses Tempo in microservices mode. For example, if you set up Tempo using the Kubernetes with Tanka procedure, then you can use this procedure to test your set up.

Before you begin

You’ll need:

Grafana 9.0.0 or higher
Microservice deployments require the Tempo querier URL, for example: http://query-frontend.tempo.svc.cluster.local:3200
OpenTelemetry telemetrygen for generating tracing data

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Set up `remote_write` to your Tempo cluster

To enable writes to your cluster:

Add a remote_write configuration snippet to the configuration file of an existing Grafana Agent.

If you do not have an existing traces collector, refer to Set up with Grafana Agent. For Kubernetes, refer to the Grafana Agent Traces Kubernetes quick start guide.

The example agent Kubernetes ConfigMap configuration below opens many trace receivers (note that the remote write is onto the Tempo cluster using OTLP gRPC):

kind: ConfigMap
metadata:
  name: grafana-agent-traces
apiVersion: v1
data:
  agent.yaml: |
    traces:
        configs:
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
            receivers:
                jaeger:
                    protocols:
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                opencensus: null
                otlp:
                    protocols:
                        grpc: null
                        http: null
                zipkin: null
            remote_write:
              - endpoint: <tempoDistributorServiceEndpoint>
                insecure: true  # only add this if TLS is not used
            scrape_configs:
              - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                job_name: kubernetes-pods
                kubernetes_sd_configs:
                  - role: pod
                relabel_configs:
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                    insecure_skip_verify: false

If you have followed the Tanka Tempo installation example, then the endpoint value would be:

distributor.tempo.svc.cluster.local:4317

Apply the ConfigMap with:

kubectl apply --namespace default -f agent.yaml

Deploy Grafana Agent using the procedures from the relevant instructions above.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

Navigate to Configuration ≫ Data Sources.
Click on Add data source.
Select Tempo.
Set the URL to http://<TEMPO-HOST>:<HTTP-LISTEN-PORT>/, filling in the path to your gateway and the configured HTTP API prefix. If you have followed the Tanka Tempo installation example, this will be: http://query-frontend.tempo.svc.cluster.local:3200/
Click Save & Test.

You should see a message that says Data source is working.

If you see an error that says Data source is not working: failed to get trace with id: 0, check your Grafana version.

To fix the error, upgrade your Grafana to 9.0 or later.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Use OpenTelemetry `telemetrygen` to generate tracing data

Next, you can use OpenTelemetry telemetrygen to generate tracing data to test your Tempo installation.

In the following instructions we assume the endpoints for both the Grafana Agent and the Tempo distributor are those described above, for example:

grafana-agent-traces.default.svc.cluster.local for Grafana Agent
distributor.tempo.svc.cluster.local for the Tempo distributor Replace these appropriately if you have altered the endpoint targets.

Install telemetrygen using the installation procedure. NOTE: You do not need to configure an OpenTelemetry Collector as we are using the Grafana Agent.

Generate traces using telemtrygen:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s grafana-agent-traces.default.svc.cluster.local:4317

This configuration sends traces to Grafana Agent for 5 seconds, at a rate of 20 traces per second.

Optionally, you can also send the trace directly to the Tempo database without using Grafana Agent as a collector by using the following:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s distributor.tempo.svc.cluster.local:4317

View tracing data in Grafana

To view the tracing data:

Go to Grafana and select the Explore menu item.
Select the Tempo data source from the list of data sources.
Copy the trace ID into the Trace ID edit field.
Select Run query.
Confirm that traces are displayed in the traces Explore panel. You should see 5 seconds worth of traces, 100 traces in total per run of telemetrygen.