Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

DocumentationTempo documentationSet upSet up a test application for a Tempo cluster
Open source

Set up a test application for a Tempo cluster

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana. This procedure uses Tempo in microservices mode. For example, if you set up Tempo using the Kubernetes with Tanka procedure, then you can use this procedure to test your set up.

Before you begin

You’ll need:

  • Grafana 9.0.0 or higher
  • Microservice deployments require the Tempo querier URL, for example: http://query-frontend.tempo.svc.cluster.local:3200
  • OpenTelemetry telemetrygen for generating tracing data

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Set up remote_write to your Tempo cluster

To enable writes to your cluster:

  1. Add a remote_write configuration snippet to the configuration file of an existing Grafana Agent.

    If you do not have an existing traces collector, refer to Set up with Grafana Agent. For Kubernetes, refer to the Grafana Agent Traces Kubernetes quick start guide.

    The example agent Kubernetes ConfigMap configuration below opens many trace receivers (note that the remote write is onto the Tempo cluster using OTLP gRPC):

    kind: ConfigMap
      name: grafana-agent-traces
    apiVersion: v1
      agent.yaml: |
              - batch:
                    send_batch_size: 1000
                    timeout: 5s
                name: default
                            grpc: null
                            thrift_binary: null
                            thrift_compact: null
                            thrift_http: null
                    opencensus: null
                            grpc: null
                            http: null
                    zipkin: null
                  - endpoint: <tempoDistributorServiceEndpoint>
                    insecure: true  # only add this if TLS is not used
                  - bearer_token_file: /var/run/secrets/
                    job_name: kubernetes-pods
                      - role: pod
                      - action: replace
                          - __meta_kubernetes_namespace
                        target_label: namespace
                      - action: replace
                          - __meta_kubernetes_pod_name
                        target_label: pod
                      - action: replace
                          - __meta_kubernetes_pod_container_name
                        target_label: container
                        ca_file: /var/run/secrets/
                        insecure_skip_verify: false

    If you have followed the Tanka Tempo installation example, then the endpoint value would be:

  2. Apply the ConfigMap with:

    kubectl apply --namespace default -f agent.yaml
  3. Deploy Grafana Agent using the procedures from the relevant instructions above.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

  1. Navigate to Configuration ≫ Data Sources.

  2. Click on Add data source.

  3. Select Tempo.

  4. Set the URL to http://<TEMPO-HOST>:<HTTP-LISTEN-PORT>/, filling in the path to your gateway and the configured HTTP API prefix. If you have followed the Tanka Tempo installation example, this will be: http://query-frontend.tempo.svc.cluster.local:3200/

  5. Click Save & Test.

You should see a message that says Data source is working.

If you see an error that says Data source is not working: failed to get trace with id: 0, check your Grafana version.

To fix the error, upgrade your Grafana to 9.0 or later.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Use OpenTelemetry telemetrygen to generate tracing data

Next, you can use OpenTelemetry telemetrygen to generate tracing data to test your Tempo installation.

In the following instructions we assume the endpoints for both the Grafana Agent and the Tempo distributor are those described above, for example:

  • grafana-agent-traces.default.svc.cluster.local for Grafana Agent
  • distributor.tempo.svc.cluster.local for the Tempo distributor Replace these appropriately if you have altered the endpoint targets.
  1. Install telemetrygen using the installation procedure. NOTE: You do not need to configure an OpenTelemetry Collector as we are using the Grafana Agent.

  2. Generate traces using telemtrygen:

    telemetrygen traces --otlp-insecure --rate 20 --duration 5s grafana-agent-traces.default.svc.cluster.local:4317

This configuration sends traces to Grafana Agent for 5 seconds, at a rate of 20 traces per second.

Optionally, you can also send the trace directly to the Tempo database without using Grafana Agent as a collector by using the following:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s distributor.tempo.svc.cluster.local:4317

View tracing data in Grafana

To view the tracing data:

  1. Go to Grafana and select the Explore menu item.

  2. Select the Tempo data source from the list of data sources.

  3. Copy the trace ID into the Trace ID edit field.

  4. Select Run query.

  5. Confirm that traces are displayed in the traces Explore panel. You should see 5 seconds worth of traces, 100 traces in total per run of telemetrygen.