Set up a Tempo clusterSet up a test app for a Tempo cluster

Set up a test application for a Tempo cluster

Once you’ve set up a Grafana Tempo cluster, you need to write some traces to it and then query the traces from within Grafana.

Before you begin

You’ll need:

  • Grafana 9.0.0 or higher
  • Microservice deployments require the Tempo querier URL, for example: http://query-frontend.tempo.svc.cluster.local:3200

Refer to Deploy Grafana on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Set up remote_write to your Tempo cluster

To enable writes to your cluster:

  1. Add a remote_write configuration snippet to the configuration file of an existing Grafana Agent.

    If you do not have an existing traces collector, refer to Set up with Grafana Agent. For Kubernetes, refer to the Grafana Agent Traces Kubernetes quick start guide.

    The example agent Kubernetes ConfigMap configuration below opens many trace receivers (note that the remote write is onto the Tempo cluster using OTLP gRPC):

    kind: ConfigMap
    metadata:
      name: grafana-agent-traces
    apiVersion: v1
    data:
      agent.yaml: |
        traces:
            configs:
              - batch:
                    send_batch_size: 1000
                    timeout: 5s
                name: default
                receivers:
                    jaeger:
                        protocols:
                            grpc: null
                            thrift_binary: null
                            thrift_compact: null
                            thrift_http: null
                    opencensus: null
                    otlp:
                        protocols:
                            grpc: null
                            http: null
                    zipkin: null
                remote_write:
                  - endpoint: <tempoDistributorServiceEndpoint>
                    insecure: true  # only add this if TLS is not used
                scrape_configs:
                  - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                    job_name: kubernetes-pods
                    kubernetes_sd_configs:
                      - role: pod
                    relabel_configs:
                      - action: replace
                        source_labels:
                          - __meta_kubernetes_namespace
                        target_label: namespace
                      - action: replace
                        source_labels:
                          - __meta_kubernetes_pod_name
                        target_label: pod
                      - action: replace
                        source_labels:
                          - __meta_kubernetes_pod_container_name
                        target_label: container
                    tls_config:
                        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                        insecure_skip_verify: false
    

    If you have followed the Tanka Tempo installation example, then the endpoint value would be:

    distributor.tempo.svc.cluster.local:4317
    
  2. Apply the ConfigMap with:

    kubectl apply --namespace default -f agent.yaml
    
  3. Deploy Grafana Agent using the procedures from the relevant instructions above.

Create a Grafana Tempo data source

To allow Grafana to read traces from Tempo, you must create a Tempo data source.

  1. Navigate to Configuration ≫ Data Sources.

  2. Click on Add data source.

  3. Select Tempo.

  4. Set the URL to http://<TEMPO-HOST>:<HTTP-LISTEN-PORT>/, filling in the path to your gateway and the configured HTTP API prefix. If you have followed the Tanka Tempo installation example, this will be: http://query-frontend.tempo.svc.cluster.local:3200/

  5. Click Save & Test.

You should see a message that says Data source is working.

If you see an error that says Data source is not working: failed to get trace with id: 0, check your Grafana version.

To fix the error, upgrade your Grafana to 9.0 or later.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page. For more information, refer to Tempo in Grafana.

Test your configuration using the TNS application

You can use The New Stack (TNS) application to test Tempo data.

  1. Create a new directory to store the TNS manifests.

  2. Navigate to https://github.com/grafana/tns/tree/main/production/k8s-yamls to get the Kubernetes manifests for the TNS application.

  3. Clone the repository using commands similar to the ones below (where <targetDir> is the directory you used to store the manifests):

      mkdir ~/tmp
      cd ~/tmp
      git clone git+ssh://github.com/grafana/tns
      cp tns/production/k8s-yamls/* <targetDir>
    
  4. Change to the new directory: cd <targetDir> .

  5. In each of the -dep.yaml manifests, alter the JAEGER_AGENT_HOST to the Grafana Agent location. For example, based on the above Grafana Agent install:

    env:
    - name: JAEGER_AGENT_HOST
      value: grafana-agent-traces.default.svc.cluster.local
    
  6. Deploy the TNS application. It will deploy into the default namespace.

      kubectl apply -f app-svc.yaml,db-svc.yaml,loadgen-svc.yaml,app-dep.yaml,db-dep.yaml,loadgen-dep.yaml
    
  7. Once the application is running, look at the logs for one of the pods (such as the App pod) and find a relevant trace ID. For example:

     kubectl logs $(kubectl get pod -l name=app -o jsonpath="{.items[0].metadata.name}")
     level=debug traceID=50075ac8b434e8f7 msg="GET / (200) 1.950625ms"
     level=info msg="HTTP client success" status=200 url=http://db duration=1.297806ms traceID=2c2fd669c388e76
     level=debug traceID=2c2fd669c388e76 msg="GET / (200) 1.70755ms"
     level=info msg="HTTP client success" status=200 url=http://db duration=1.853271ms traceID=79058bb9cc39acfb
     level=debug traceID=79058bb9cc39acfb msg="GET / (200) 2.300922ms"
     level=info msg="HTTP client success" status=200 url=http://db duration=1.381894ms traceID=7b0e0526f5958549
     level=debug traceID=7b0e0526f5958549 msg="GET / (200) 2.105263ms"
    
  8. Go to Grafana and select the Explore menu item.

  9. Select the Tempo data source from the list of data sources.

  10. Copy the trace ID into the Trace ID edit field.

  11. Select Run query.

  12. Confirm that the trace is displayed in the traces Explore panel.