Validate deployment on Grafana Labs

Validate your local Tempo deployment

Thu, 09 Apr 2026 14:59:14 +0000

Validate your local Tempo deployment

Once you’ve set up Grafana Tempo, the next step is to test your deployment to ensure that traces are emitted and collected correctly. This procedure uses a Docker Compose example in the Tempo repository.

Verify your cluster is working

To verify that Tempo is working, run the following command:

systemctl is-active tempo

You should see the status active returned. If you don’t, check that the configuration file is correct, and then restart the service. You can also use journalctl -u tempo to view the logs for Tempo to determine if there are any obvious reasons for failure to start.

Verify that your storage bucket has received data by signing in to your storage provider and determining that a file has been written to storage. It should be called tempo_cluster_seed.json.

Test your installation

Once Tempo is running, you can use the K6 with Traces Docker example to verify that trace data is sent to Tempo. This procedure sets up a sample data source in Grafana to read from Tempo.

Backend storage configuration

The Tempo examples running with docker-compose all include a version of Tempo and a storage backend like S3 and GCS. Because Tempo is installed with a backend storage configured, you need to change the docker-compose.yaml file to remove Tempo and instead point trace storage to the installed version. These steps are included in this section.

Network configuration

Docker compose uses an internal networking bridge to connect all of the defined services. Because the Tempo instance is running as a service on the local machine host, you need the resolvable IP address of the local machine so the docker containers can use the Tempo service. You can find the host IP address of your Linux machine using a command such as ip addr show.

Steps

Clone the Tempo repository:

git clone https://github.com/grafana/tempo.git

Go into the examples directory:
```
cd tempo/example/docker-compose/local
```
Edit the file docker-compose.yaml, and remove the tempo service and all its properties, so that the first service defined is k6-tracing. The start of your docker-compose.yaml should look like this:
```
version: "3"
services:

k6-tracing:
```
Edit the k6-tracing service, and change the value of ENDPOINT to the local IP address of the machine running Tempo and docker compose, eg. 10.128.0.104:4317. This is the OTLP gRPC port:
```
environment:
  - ENDPOINT=10.128.0.104:4317
```
This ensures that the traces sent from the example application go to the locally running Tempo service on the Linux machine.
Edit the k6-tracing service and remove the dependency on Tempo by deleting the following lines:
```
depends_on:
tempo
```
Save the docker-compose.yaml file and exit your editor.
Edit the default Grafana data source for Tempo that is included in the examples. Edit the file located at tempo/example/shared/grafana-datasources.yaml, and change the url field of the Tempo data source to point to the local IP address of the machine running the Tempo service instead (eg. url: http://10.128.0.104:3200). The Tempo data source section should resemble this:
```
- name: Tempo
  type: tempo
  access: proxy
  orgId: 1
  url: http://10.128.0.104:3200
```
Save the file and exit your editor.
Edit the Prometheus configuration file so it uses the Tempo service as a scrape target. Change the target to the local Linux host IP address. Edit the tempo/example/shared/prometheus.yaml file, and alter the tempo job to replace tempo:3200 with the Linux machine host IP address.
YAML
```
  - job_name: 'tempo'
	static_configs:
  	- targets: [ '10.128.0.104:3200' ]
```
Save the file and exit your editor.**
Start the three services that are defined in the docker-compose file:
Bash
```
docker compose up -d
```

Verify that the services are running using docker compose ps. You should see something like:

NAME             	IMAGE                                   	COMMAND              	SERVICE         	CREATED         	STATUS          	PORTS
local-grafana-1  	grafana/grafana:9.3.2                   	"/run.sh"            	grafana         	2 minutes ago   	Up 3 seconds    	0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
local-k6-tracing-1   ghcr.io/grafana/xk6-client-tracing:v0.0.9   "/k6-tracing run /ex…"   k6-tracing      	2 minutes ago   	Up 2 seconds
local-prometheus-1   prom/prometheus:latest                  	"/bin/prometheus --c…"   prometheus      	2 minutes ago   	Up 2 seconds    	0.0.0.0:9090->9090/tcp, :::9090->9090/tcp

Grafana is running on port 3000, Prometheus is running on port 9090. Both should be bound to the host machine.

As part of the docker compose manifest, Grafana is now running on your Linux machine, reachable on port 3000. Point your web browser to the Linux machine on port 3000. You might need to port forward the local port if you’re doing this remotely, for example, via SSH forwarding.
Once logged in, navigate to the Explore page, select the Tempo data source and select the Search tab. Select Run query to list the recent traces stored in Tempo. Select one to view the trace diagram:
Alter the Tempo configuration to point to the instance of Prometheus running in docker compose. To do so, edit the configuration at /etc/tempo/config.yaml and change the storage block under the metrics_generator section so that the remote write URL is http://localhost:9090. The configuration section should look like this:
YAML
```
 storage:
     path: /var/tempo/generator/wal
     remote_write:
        - url: http://localhost:9090/api/v1/write
        send_exemplars: true
```
Save the file and exit the editor.
Finally, restart the Tempo service by running:
```
sudo systemctl restart tempo
```
A couple of minutes after Tempo has successfully restarted, select the Service graph tab for the Tempo data source in the Explore page. Select Run query to view a service graph, generated by Tempo’s metrics-generator.

Validate Kubernetes deployment using a test application

Thu, 09 Apr 2026 14:59:14 +0000

Validate Kubernetes deployment using a test application

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana. This procedure uses Tempo in microservices mode. For example, if you set up Tempo using the Kubernetes with Tanka procedure, then you can use this procedure to validate your set up.

Before you begin

You’ll need:

Grafana 10.0.0 or higher
Microservice deployments require the Tempo querier URL, for example: http://tempo-cluster-query-frontend.tempo.svc.cluster.local:3100/
OpenTelemetry telemetrygen for generating tracing data

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Configure Grafana Alloy to remote-write to Tempo

Note
You can skip this section if you have already configured Alloy to send traces to Tempo.

This section uses a Grafana Alloy Helm chart deployment to send traces to Tempo.

To do this, you need to create a configuration that can be used by Alloy to receive and export traces in OTLP protobuf format.

Create a new values.yaml file which we’ll use as part of the Alloy install.

Edit the values.yaml file and add the following configuration to it:

alloy:
  extraPorts:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
      protocol: TCP
  configMap:
    create: true
    content: |-
      // Creates a receiver for OTLP gRPC.
      // You can easily add receivers for other protocols by using the correct component
      // from the reference list at: https://grafana.com/docs/alloy/latest/reference/components/
      otelcol.receiver.otlp "otlp_receiver" {
        // Listen on all available bindable addresses on port 4317 (which is the
        // default OTLP gRPC port) for the OTLP protocol.
        grpc {
          endpoint = "0.0.0.0:4317"
        }

        // Output straight to the OTLP gRPC exporter. We would usually do some processing
        // first, most likely batch processing, but for this example we pass it straight
        // through.
        output {
          traces = [
            otelcol.exporter.otlp.tempo.input,
          ]
        }
      }

      // Define an OTLP gRPC exporter to send all received traces to GET.
      // The unique label 'tempo' is added to uniquely identify this exporter.
      otelcol.exporter.otlp "tempo" {
          // Define the client for exporting.
          client {
              // Send to the locally running Tempo instance, on port 4317 (OTLP gRPC).
              endpoint = "tempo-cluster-distributor.tempo.svc.cluster.local:4317"
              // Disable TLS for OTLP export.
              tls {
                  // The connection is insecure.
                  insecure = true
                  // Do not verify TLS certificates when connecting.
                  insecure_skip_verify = true
              }
          }
      }

Ensure that you use the specific namespace you’ve installed Tempo in for the OTLP exporter. In the line:

endpoint = "tempo-cluster-distributor.tempo.svc.cluster.local:4317"

change tempo to reference the namespace where Tempo is installed, for example: tempo-cluster-distributor.my-tempo-namespace.svc.cluster.local:4317.

Deploy Alloy using Helm:
Bash
```
helm install -f values.yaml grafana-alloy grafana/alloy
```
If you deploy Alloy into a specific namespace, create the namespace first and specify it to Helm by appending --namespace=<grafana-alloy-namespace> to the end of the command.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

Navigate to Connections > Data Sources.
Click on Add data source.
Select Tempo.
Set the URL to http://<TEMPO-QUERY-FRONTEND-SERVICE>:<HTTP-LISTEN-PORT>/, filling in the path to the Tempo query frontend service and the configured HTTP API prefix. If you followed Deploy Tempo with Helm installation example, the query frontend service’s URL looks something like this: http://tempo-cluster-query-frontend.<NAMESPACE>.svc.cluster.local:3100
Click Save & Test.

You should see a message that says Data source is working.

Visualize your data

After you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Use OpenTelemetry `telemetrygen` to generate tracing data

You can use OpenTelemetry telemetrygen to generate tracing data to test your Tempo installation.

These instructions use the endpoints for both Grafana Alloy and the Tempo distributor used previously, for example:

grafana-alloy.grafana-alloy.svc.cluster.local for Grafana Alloy
tempo-cluster-distributor.tempo.svc.cluster.local for the Tempo distributor

Update the endpoints if you have altered the endpoint targets.

Install telemetrygen using the installation procedure. NOTE: You don’t need to configure an OpenTelemetry Collector because we’re using Grafana Alloy.
Generate traces using telemetrygen:
Bash
```
telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint grafana-alloy.grafana-alloy.svc.cluster.local:4317
```
This configuration sends traces to Alloy for 5 seconds, at a rate of 20 traces per second.

Optionally, you can also send the trace directly to the Tempo database without using Alloy as a collector by using the following:

telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint tempo-cluster-distributor.tempo.svc.cluster.local:4317

If you’re running telemetrygen on your local machine, ensure that you first port-forward to the relevant Alloy or Tempo distributor service, for example:

kubectl port-forward services/grafana-alloy 4317:4317 --namespace grafana-alloy

Alternatively, you can create a cronjob to send traces periodically based on this template:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: sample-traces
spec:
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 2
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      backoffLimit: 0
      ttlSecondsAfterFinished: 3600
      template:
        spec:
          containers:
          - name: traces
            image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.96.0
            args:
              - traces
              - --otlp-insecure
              - --rate
              - "20"
              - --duration
              - 5s
              - --otlp-endpoint
              - grafana-alloy.grafana-alloy.svc.cluster.local:4317
          restartPolicy: Never

To view the tracing data:

Go to Grafana and select Explore.
Select the Tempo data source from the list of data sources.
Select the Search Query type.
Select Run query.
Confirm that traces are displayed in the traces Explore panel. You should see 5 seconds worth of traces, 100 traces in total per run of telemetrygen.

Test your configuration using the Intro to MLTP application

The Intro to MLTP application provides an example five-service application generates data for Tempo, Mimir, Loki, and Pyroscope. This procedure installs the application on your cluster so you can generate meaningful test data.

Navigate to https://github.com/grafana/intro-to-mltp to get the Kubernetes manifests for the Intro to MLTP application.

Clone the repository using commands similar to the ones below:

  git clone git+ssh://github.com/grafana/intro-to-mltp
  cp intro-to-mltp/k8s/mythical/* ~/tmp/intro-to-mltp-k8s

Change to the cloned repository: cd intro-to-mltp/k8s/mythical
In the mythical-beasts-deployment.yaml manifest, alter each TRACING_COLLECTOR_HOST environment variable instance value to point to the Grafana Alloy location. For example, based on Alloy installed in the default namespace and with a Helm installation called test:
YAML
```
 	- env:
     ...
     - name: TRACING_COLLECTOR_HOST
       value: grafana-alloy.grafana-alloy.svc.cluster.local
```

Deploy the Intro to MLTP application. It deploys into the default namespace.

   kubectl apply -f mythical-beasts-service.yaml,mythical-beasts-persistentvolumeclaim.yaml,mythical-beasts-deployment.yaml

Once the application is deployed, go to Grafana and select the Explore menu item.
Select the Tempo data source from the list of data sources.
Select the Search Query type for the data source.
Select Run query.
Traces from the application are displayed in the traces Explore panel.

Validate deployment on Grafana Labs

Validate your local Tempo deployment

Validate your local Tempo deployment

Verify your cluster is working

Test your installation

Backend storage configuration

Network configuration

Steps

Validate Kubernetes deployment using a test application

Validate Kubernetes deployment using a test application

Before you begin

Configure Grafana Alloy to remote-write to Tempo

Create a Grafana Tempo data source

Visualize your data

Use OpenTelemetry telemetrygen to generate tracing data

Test your configuration using the Intro to MLTP application

Use OpenTelemetry `telemetrygen` to generate tracing data