Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation

Tempo documentation

Set up a Tempo server or cluster

Set up a test application for a Tempo cluster

Open source

Set up a test application for a Tempo cluster

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana.

Before you begin

You’ll need:

Grafana 9.0.0 or higher
Microservice deployments require the Tempo querier URL, for example: http://query-frontend.tempo.svc.cluster.local:3200

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Set up `remote_write` to your Tempo cluster

To enable writes to your cluster:

Add a remote_write configuration snippet to the configuration file of an existing Grafana Agent.

If you do not have an existing traces collector, refer to Set up with Grafana Agent. For Kubernetes, refer to the Grafana Agent Traces Kubernetes quick start guide.

The example agent Kubernetes ConfigMap configuration below opens many trace receivers (note that the remote write is onto the Tempo cluster using OTLP gRPC):

kind: ConfigMap
metadata:
  name: grafana-agent-traces
apiVersion: v1
data:
  agent.yaml: |
    traces:
        configs:
          - batch:
                send_batch_size: 1000
                timeout: 5s
            name: default
            receivers:
                jaeger:
                    protocols:
                        grpc: null
                        thrift_binary: null
                        thrift_compact: null
                        thrift_http: null
                opencensus: null
                otlp:
                    protocols:
                        grpc: null
                        http: null
                zipkin: null
            remote_write:
              - endpoint: <tempoDistributorServiceEndpoint>
                insecure: true  # only add this if TLS is not used
            scrape_configs:
              - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                job_name: kubernetes-pods
                kubernetes_sd_configs:
                  - role: pod
                relabel_configs:
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                    insecure_skip_verify: false

If you have followed the Tanka Tempo installation example, then the endpoint value would be:

distributor.tempo.svc.cluster.local:4317

Apply the ConfigMap with:

kubectl apply --namespace default -f agent.yaml

Deploy Grafana Agent using the procedures from the relevant instructions above.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

Navigate to Configuration ≫ Data Sources.
Click on Add data source.
Select Tempo.
Set the URL to http://<TEMPO-HOST>:<HTTP-LISTEN-PORT>/, filling in the path to your gateway and the configured HTTP API prefix. If you have followed the Tanka Tempo installation example, this will be: http://query-frontend.tempo.svc.cluster.local:3200/
Click Save & Test.

You should see a message that says Data source is working.

If you see an error that says Data source is not working: failed to get trace with id: 0, check your Grafana version.

To fix the error, upgrade your Grafana to 9.0 or later.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Test your configuration using the TNS application

You can use The New Stack (TNS) application to test Tempo data.

Create a new directory to store the TNS manifests.
Navigate to https://github.com/grafana/tns/tree/main/production/k8s-yamls to get the Kubernetes manifests for the TNS application.
Clone the repository using commands similar to the ones below (where <targetDir> is the directory you used to store the manifests):
bash
```
  mkdir ~/tmp
  cd ~/tmp
  git clone git+ssh://github.com/grafana/tns
  cp tns/production/k8s-yamls/* <targetDir>
```
Change to the new directory: cd <targetDir> .
In each of the -dep.yaml manifests, alter the JAEGER_AGENT_HOST to the Grafana Agent location. For example, based on the above Grafana Agent install:
yaml
```
env:
- name: JAEGER_AGENT_HOST
  value: grafana-agent-traces.default.svc.cluster.local
```

Deploy the TNS application. It will deploy into the default namespace.

  kubectl apply -f app-svc.yaml,db-svc.yaml,loadgen-svc.yaml,app-dep.yaml,db-dep.yaml,loadgen-dep.yaml

Once the application is running, look at the logs for one of the pods (such as the App pod) and find a relevant trace ID. For example:

 kubectl logs $(kubectl get pod -l name=app -o jsonpath="{.items[0].metadata.name}")
 level=debug traceID=50075ac8b434e8f7 msg="GET / (200) 1.950625ms"
 level=info msg="HTTP client success" status=200 url=http://db duration=1.297806ms traceID=2c2fd669c388e76
 level=debug traceID=2c2fd669c388e76 msg="GET / (200) 1.70755ms"
 level=info msg="HTTP client success" status=200 url=http://db duration=1.853271ms traceID=79058bb9cc39acfb
 level=debug traceID=79058bb9cc39acfb msg="GET / (200) 2.300922ms"
 level=info msg="HTTP client success" status=200 url=http://db duration=1.381894ms traceID=7b0e0526f5958549
 level=debug traceID=7b0e0526f5958549 msg="GET / (200) 2.105263ms"