Validate deployment on Grafana Labs

Validate your local Tempo deployment

Thu, 28 May 2026 17:50:33 +0100

Validate your local Tempo deployment

After you’ve set up Grafana Tempo, the next step is to verify that traces are ingested, stored, and queryable.

This page covers two levels of validation:

Quick validation uses telemetrygen and the Tempo HTTP API to confirm traces flow end-to-end from the command line, with no additional services required.
Grafana validation connects Grafana to your local Tempo instance so you can explore traces visually.

Before you begin

To follow this procedure, you need:

A locally installed and running Tempo service (refer to Deploy on Linux)
OpenTelemetry telemetrygen installed for generating test traces
A running Grafana instance (refer to the installation instructions), only required for the Grafana validation section

Verify Tempo is running

Check the service status:
Bash
```
systemctl is-active tempo
```
You should see the status active returned. If you don’t, check that the configuration file is correct, and then restart the service. You can also use journalctl -u tempo to view the logs for Tempo to determine if there are any obvious reasons for failure to start.
Verify that Tempo created the storage subdirectories:
Bash
```
ls /data/tempo/
```
You should see wal and blocks directories.

If you configured Tempo with an S3-compatible backend instead of local storage, refer to Verify data in your S3-compatible store for storage verification steps.

Send test traces

Use OpenTelemetry telemetrygen to send traces to Tempo. The following command sends 100 traces (20 per second for 5 seconds) over OTLP gRPC:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint localhost:4317

Verify traces using the Tempo API

After sending traces, you can verify them directly using the Tempo HTTP API. No additional services are required.

Search for recent traces:
Bash
```
curl -s http://localhost:3200/api/search | jq .
```
You should see a traces array containing your test traces. Each entry includes a traceID, rootServiceName, and rootTraceName. For example:
JSON
```
{
  "traces": [
    {
      "traceID": "abc123...",
      "rootServiceName": "telemetrygen",
      "rootTraceName": "lets-go",
      "startTimeUnixNano": "1776912138880042305"
    }
  ]
}
```
Note
Trace data appears in search results after the live store flushes completed blocks to disk, which can take 15–30 seconds. If the traces array is empty, wait and retry.
Retrieve a specific trace by ID using a traceID value from the search results:
Bash
```
curl -s http://localhost:3200/api/v2/traces/<TRACE_ID> | jq .
```
Replace <TRACE_ID> with an actual trace ID from the previous step.

If both commands return trace data, your Tempo installation is ingesting, storing, and serving traces correctly.

Refer to the Tempo HTTP API documentation for the full API reference and Push spans with HTTP for an alternative approach using curl to send traces.

Verify traces in Grafana

If you have a Grafana instance running, you can verify that traces are visible in the Grafana Explore view. If Grafana is not already installed, refer to the installation instructions. For the full Tempo data source configuration reference, refer to Configure the Tempo data source.

In Grafana, navigate to Connections > Data sources and select Add data source.
Select Tempo and set the URL to http://localhost:3200.
Select Save & Test. You should see a message that says Data source is working.
Navigate to the Explore page, select the Tempo data source, and choose the Search query type.
Select Run query to list the recent traces stored in Tempo. You should see traces from telemetrygen in the results. Select a trace to view its spans:

Optional: Metrics-generator

If you are using the metrics-generator, you can enable it to generate metrics from the traces.

Edit the configuration at /etc/tempo/config.yml and add or update the metrics_generator section. Set the remote_write URL to the address of your Prometheus-compatible storage instance. The configuration section should look like this:
YAML
```
metrics_generator:
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://localhost:9090/api/v1/write
        send_exemplars: true
```
Save the file and exit the editor.
Restart the Tempo service:
Bash
```
sudo systemctl restart tempo
```
After a couple of minutes, select the Service graph tab for the Tempo data source in the Explore page. Select Run query to view a service graph, generated by the Tempo metrics-generator.

Next steps

After you validate your local Tempo deployment, consider exploring these topics:

Instrument for distributed tracing to send traces from your own applications instead of a test generator
Configure Tempo to customize settings for your environment
Set up monitoring for Tempo to observe your Tempo instance with dashboards and alerts
Use tracing data in Grafana to learn more about querying and visualizing traces

Validate Kubernetes deployment using a test application

Thu, 28 May 2026 17:50:33 +0100

Validate Kubernetes deployment using a test application

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana. This procedure uses Tempo in microservices mode. For example, if you set up Tempo using the Kubernetes with Tanka procedure, then you can use this procedure to validate your set up.

Before you begin

You’ll need:

Grafana 10.0.0 or higher
Microservice deployments require the Tempo query frontend URL, for example: http://<TEMPO-QUERY-FRONTEND-SERVICE>.<NAMESPACE>.svc.cluster.local:3200/
OpenTelemetry telemetrygen for generating tracing data

Note
Service names depend on how you deployed Tempo. For Tanka deployments, services use plain names like distributor and query-frontend. For Helm deployments, services are prefixed with the release name, for example tempo-cluster-distributor. Update the service names in the examples below to match your deployment.

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Configure Grafana Alloy to remote-write to Tempo

Note
You can skip this section if you have already configured Alloy to send traces to Tempo.

This section uses a Grafana Alloy Helm chart deployment to send traces to Tempo.

To do this, you need to create a configuration that can be used by Alloy to receive and export traces in OTLP protobuf format.

Create a new values.yaml file which we’ll use as part of the Alloy install.

Edit the values.yaml file and add the following configuration to it:

alloy:
  extraPorts:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
      protocol: TCP
  configMap:
    create: true
    content: |-
      // Creates a receiver for OTLP gRPC.
      // You can easily add receivers for other protocols by using the correct component
      // from the reference list at: https://grafana.com/docs/alloy/latest/reference/components/
      otelcol.receiver.otlp "otlp_receiver" {
        // Listen on all available bindable addresses on port 4317 (which is the
        // default OTLP gRPC port) for the OTLP protocol.
        grpc {
          endpoint = "0.0.0.0:4317"
        }

        // Output straight to the OTLP gRPC exporter. We would usually do some processing
        // first, most likely batch processing, but for this example we pass it straight
        // through.
        output {
          traces = [
            otelcol.exporter.otlp.tempo.input,
          ]
        }
      }

      // Define an OTLP gRPC exporter to send all received traces to Tempo.
      // The unique label 'tempo' is added to uniquely identify this exporter.
      otelcol.exporter.otlp "tempo" {
          // Define the client for exporting.
          client {
             // Send to the Tempo distributor on port 4317 (OTLP gRPC).
             endpoint = "distributor.tempo.svc.cluster.local:4317"
              // Disable TLS for OTLP export.
              tls {
                  // The connection is insecure.
                  insecure = true
                  // Do not verify TLS certificates when connecting.
                  insecure_skip_verify = true
              }
          }
      }

Update the distributor endpoint to match your deployment. The service name and namespace depend on how you deployed Tempo:

Tanka: distributor.<NAMESPACE>.svc.cluster.local:4317
Helm: <RELEASE-NAME>-distributor.<NAMESPACE>.svc.cluster.local:4317

Deploy Alloy using Helm:
Bash
```
helm install -f values.yaml grafana-alloy grafana/alloy
```
If you deploy Alloy into a specific namespace, create the namespace first and specify it to Helm by appending --namespace=<grafana-alloy-namespace> to the end of the command.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

Navigate to Connections > Data Sources.
Click Add data source.
Select Tempo.
Set the URL to http://<TEMPO-QUERY-FRONTEND-SERVICE>:<HTTP-LISTEN-PORT>/, filling in the path to the Tempo query frontend service and the HTTP listen port (default 3200). For example:
- Tanka deployment: http://query-frontend.tempo.svc.cluster.local:3200
- Helm deployment: http://tempo-cluster-query-frontend.<NAMESPACE>.svc.cluster.local:3200
Click Save & Test.

You should see a message that says Data source is working.

Visualize your data

After you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Use OpenTelemetry `telemetrygen` to generate tracing data

You can use OpenTelemetry telemetrygen to generate tracing data to test your Tempo installation.

These instructions use the endpoints for both Grafana Alloy and the Tempo distributor, for example:

grafana-alloy.grafana-alloy.svc.cluster.local for Grafana Alloy
distributor.tempo.svc.cluster.local for the Tempo distributor (Tanka deployment)

Update the endpoints to match your deployment.

Install telemetrygen using the installation procedure.

Note
You don’t need to configure an OpenTelemetry Collector because this procedure uses Grafana Alloy.
Generate traces using telemetrygen:
Bash
```
telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint grafana-alloy.grafana-alloy.svc.cluster.local:4317
```
This configuration sends traces to Alloy for 5 seconds, at a rate of 20 traces per second.

Optionally, you can also send traces directly to Tempo without using Alloy as a collector:
Bash
```
telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint distributor.tempo.svc.cluster.local:4317
```
If you’re running telemetrygen on your local machine, ensure that you first port-forward to the relevant Alloy or Tempo distributor service, for example:
Bash
```
kubectl port-forward services/grafana-alloy 4317:4317 --namespace grafana-alloy
```

Alternatively, you can create a cronjob to send traces periodically based on this template:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: sample-traces
spec:
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 2
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      backoffLimit: 0
      ttlSecondsAfterFinished: 3600
      template:
        spec:
          containers:
          - name: traces
            image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.150.0
            args:
              - traces
              - --otlp-insecure
              - --rate
              - "20"
              - --duration
              - 5s
              - --otlp-endpoint
              - grafana-alloy.grafana-alloy.svc.cluster.local:4317
          restartPolicy: Never

To view the tracing data:

Go to Grafana and select Explore.
Select the Tempo data source from the list of data sources.
Select the Search Query type.
Select Run query.
Confirm that traces are displayed in the traces Explore panel. You should see 5 seconds worth of traces, 100 traces in total per run of telemetrygen.

Test your configuration using the Intro to MLTP application

The Intro to MLTP application provides an example five-service application that generates data for Tempo, Mimir, Loki, and Pyroscope. This procedure installs the application on your cluster so you can generate meaningful test data.

Clone the Intro to MLTP repository:

git clone https://github.com/grafana/intro-to-mltp.git
cd intro-to-mltp/k8s/mythical

In the mythical-beasts-deployment.yaml manifest, alter each TRACING_COLLECTOR_HOST environment variable instance value to point to the Grafana Alloy location. For example, based on Alloy installed in the default namespace and with a Helm installation called test:
YAML
```
 	- env:
     ...
     - name: TRACING_COLLECTOR_HOST
       value: grafana-alloy.grafana-alloy.svc.cluster.local
```

Deploy the Intro to MLTP application. It deploys into the default namespace.

   kubectl apply -f mythical-beasts-service.yaml,mythical-beasts-persistentvolumeclaim.yaml,mythical-beasts-deployment.yaml

Once the application is deployed, go to Grafana and select the Explore menu item.
Select the Tempo data source from the list of data sources.
Select the Search Query type for the data source.
Select Run query.
Traces from the application are displayed in the traces Explore panel.

Validate deployment on Grafana Labs

Validate your local Tempo deployment

Validate your local Tempo deployment

Before you begin

Verify Tempo is running

Send test traces

Verify traces using the Tempo API

Verify traces in Grafana

Optional: Metrics-generator

Next steps

Validate Kubernetes deployment using a test application

Validate Kubernetes deployment using a test application

Before you begin

Configure Grafana Alloy to remote-write to Tempo

Create a Grafana Tempo data source

Visualize your data

Use OpenTelemetry telemetrygen to generate tracing data

Test your configuration using the Intro to MLTP application

Use OpenTelemetry `telemetrygen` to generate tracing data