--- title: "Validate Kubernetes deployment using a test application | Grafana Tempo documentation" description: "Validate your Tempo deployment on Kubernetes." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Validate Kubernetes deployment using a test application Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana. This procedure uses Tempo in microservices mode. For example, if you [set up Tempo using the Kubernetes with Tanka procedure](../../deploy/kubernetes/tanka/), then you can use this procedure to validate your set up. ## Before you begin You’ll need: - Grafana 10.0.0 or higher - Microservice deployments require the Tempo query frontend URL, for example: `http://..svc.cluster.local:3200/` - [OpenTelemetry telemetrygen](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) for generating tracing data > Note > > Service names depend on how you deployed Tempo. For Tanka deployments, services use plain names like `distributor` and `query-frontend`. For Helm deployments, services are prefixed with the release name, for example `tempo-cluster-distributor`. Update the service names in the examples below to match your deployment. Refer to [Deploy Grafana on Kubernetes](/docs/grafana/next/setup-grafana/installation/kubernetes/) if you are using Kubernetes. Otherwise, refer to [Install Grafana](/docs/grafana/next/setup-grafana/installation/) for more information. ## Configure Grafana Alloy to remote-write to Tempo > Note > > You can skip this section if you have already configured Alloy to send traces to Tempo. This section uses a [Grafana Alloy Helm chart](/docs/alloy/latest/set-up/install/kubernetes/) deployment to send traces to Tempo. To do this, you need to create a configuration that can be used by Alloy to receive and export traces in OTLP `protobuf` format. 1. Create a new `values.yaml` file which we’ll use as part of the Alloy install. 2. Edit the `values.yaml` file and add the following configuration to it: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml alloy: extraPorts: - name: otlp-grpc port: 4317 targetPort: 4317 protocol: TCP configMap: create: true content: |- // Creates a receiver for OTLP gRPC. // You can easily add receivers for other protocols by using the correct component // from the reference list at: https://grafana.com/docs/alloy/latest/reference/components/ otelcol.receiver.otlp "otlp_receiver" { // Listen on all available bindable addresses on port 4317 (which is the // default OTLP gRPC port) for the OTLP protocol. grpc { endpoint = "0.0.0.0:4317" } // Output straight to the OTLP gRPC exporter. We would usually do some processing // first, most likely batch processing, but for this example we pass it straight // through. output { traces = [ otelcol.exporter.otlp.tempo.input, ] } } // Define an OTLP gRPC exporter to send all received traces to GET. // The unique label 'tempo' is added to uniquely identify this exporter. otelcol.exporter.otlp "tempo" { // Define the client for exporting. client { // Send to the locally running Tempo instance, on port 4317 (OTLP gRPC). endpoint = "http://tempo-cluster-distributor.tempo.svc.cluster.local:4317" // Disable TLS for OTLP remote write. tls { // The connection is insecure. insecure = true // Do not verify TLS certificates when connecting. insecure_skip_verify = true } } } ``` Ensure that you use the specific namespace you’ve installed Tempo in for the OTLP exporter. In the line: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml endpoint = "http://tempo-cluster-distributor.tempo.svc.cluster.local:3100" ``` change `tempo` to reference the namespace where Tempo is installed, for example: `http://tempo-cluster-distributor.my-tempo-namespaces.svc.cluster.local:3100`. 3. Deploy Alloy using Helm: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm install -f values.yaml grafana-alloy grafana/alloy ``` If you deploy Alloy into a specific namespace, create the namespace first and specify it to Helm by appending `--namespace=` to the end of the command. ## Create a Grafana Tempo data source To allow Grafana to read traces from Tempo, you must create a Tempo data source. 1. Navigate to **Connections** > **Data Sources**. 2. Click **Add data source**. 3. Select **Tempo**. 4. Set the URL to `http://:/`, filling in the path to the Tempo query frontend service and the HTTP listen port (default `3200`). For example: - Tanka deployment: `http://query-frontend.tempo.svc.cluster.local:3200` - Helm deployment: `http://tempo-cluster-query-frontend..svc.cluster.local:3200` 5. Click **Save & Test**. You should see a message that says `Data source is working`. ## Visualize your data After you have created a data source, you can visualize your traces in the **Grafana Explore** page. For more information, refer to [Tempo in Grafana](/docs/tempo/next/introduction/tempo-in-grafana/). ### Use OpenTelemetry `telemetrygen` to generate tracing data You can use [OpenTelemetry `telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) to generate tracing data to test your Tempo installation. These instructions use the endpoints for both Grafana Alloy and the Tempo distributor, for example: - `grafana-alloy.grafana-alloy.svc.cluster.local` for Grafana Alloy - `distributor.tempo.svc.cluster.local` for the Tempo distributor (Tanka deployment) Update the endpoints to match your deployment. 1. Install `telemetrygen` using the [installation procedure](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen). > Note > > You don’t need to configure an OpenTelemetry Collector because this procedure uses Grafana Alloy. 2. Generate traces using `telemetrygen`: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint grafana-alloy.grafana-alloy.svc.cluster.local:4317 ``` This configuration sends traces to Alloy for 5 seconds, at a rate of 20 traces per second. Optionally, you can also send traces directly to Tempo without using Alloy as a collector: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint distributor.tempo.svc.cluster.local:4317 ``` If you’re running `telemetrygen` on your local machine, ensure that you first port-forward to the relevant Alloy or Tempo distributor service, for example: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl port-forward services/grafana-alloy 4317:4317 --namespace grafana-alloy ``` 3. Alternatively, you can create a cronjob to send traces periodically based on this template: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none apiVersion: batch/v1 kind: CronJob metadata: name: sample-traces spec: concurrencyPolicy: Forbid successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 2 schedule: "0 * * * *" jobTemplate: spec: backoffLimit: 0 ttlSecondsAfterFinished: 3600 template: spec: containers: - name: traces image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.150.0 args: - traces - --otlp-insecure - --rate - "20" - --duration - 5s - --otlp-endpoint - grafana-alloy.grafana-alloy.svc.cluster.local:4317 restartPolicy: Never ``` To view the tracing data: 1. Go to Grafana and select **Explore**. 2. Select the **Tempo data source** from the list of data sources. 3. Select the `Search` Query type. 4. Select **Run query**. 5. Confirm that traces are displayed in the traces **Explore** panel. You should see 5 seconds worth of traces, 100 traces in total per run of `telemetrygen`. ### Test your configuration using the Intro to MLTP application The Intro to MLTP application provides an example five-service application that generates data for Tempo, Mimir, Loki, and Pyroscope. This procedure installs the application on your cluster so you can generate meaningful test data. 1. Clone the [Intro to MLTP repository](https://github.com/grafana/intro-to-mltp): Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash git clone https://github.com/grafana/intro-to-mltp.git cd intro-to-mltp/k8s/mythical ``` 2. In the `mythical-beasts-deployment.yaml` manifest, alter each `TRACING_COLLECTOR_HOST` environment variable instance value to point to the Grafana Alloy location. For example, based on Alloy installed in the default namespace and with a Helm installation called `test`: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml - env: ... - name: TRACING_COLLECTOR_HOST value: grafana-alloy.grafana-alloy.svc.cluster.local ``` 3. Deploy the Intro to MLTP application. It deploys into the default namespace. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f mythical-beasts-service.yaml,mythical-beasts-persistentvolumeclaim.yaml,mythical-beasts-deployment.yaml ``` 4. Once the application is deployed, go to Grafana and select the **Explore** menu item. 5. Select the **Tempo data source** from the list of data sources. 6. Select the `Search` Query type for the data source. 7. Select **Run query**. 8. Traces from the application are displayed in the traces **Explore** panel.