Sometimes it can be difficult to tell what, if anything, the Grafana Agent is sending along to the backend. This document focuses on a few techniques to gain visibility on how many traces are being pushed to the Agent and if they’re making it to the backend. The tracing pipeline is built on top of the OpenTelemetry Collector which does a fantastic job of logging network and other issues.
If your logs are showing no obvious errors try the following:
The agent publishes a few Prometheus metrics that are useful to determine how much trace traffic it is receiving and successfully forwarding. These are a good place to start when diagnosing tracing Agent issues.
tempo_receiver_accepted_spans tempo_receiver_refused_spans tempo_exporter_sent_spans tempo_exporter_send_failed_spans
If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud then we can turn on Automatic Logging. A recommend debug setup is:
tempo: configs: - name: default ... automatic_logging: backend: stdout roots: true
This will emit logs to stdout for every root span that the Agent forwards. This can be useful to see exactly which traces are being forwarded to Grafana Cloud.
Related Tempo resources
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.