TroubleshootingAgent Troubleshooting

Agent Troubleshooting

Sometimes it can be difficult to tell what, if anything, the Grafana Agent is sending along to the backend. This document focuses on a few techniques to gain visibility on how many traces are being pushed to the Agent and if they’re making it to the backend. The tracing pipeline is built on top of the OpenTelemetry Collector which does a fantastic job of logging network and other issues.

If your logs are showing no obvious errors try the following:

Metrics

The agent publishes a few Prometheus metrics that are useful to determine how much trace traffic it is receiving and successfully forwarding. These are a good place to start when diagnosing tracing Agent issues.

tempo_receiver_accepted_spans
tempo_receiver_refused_spans
tempo_exporter_sent_spans
tempo_exporter_send_failed_spans

Automatic Logging

If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud then we can turn on Automatic Logging. A recommend debug setup is:

tempo:
  configs:
  - name: default
    ...
    automatic_logging:
      backend: stdout
      roots: true

This will emit logs to stdout for every root span that the Agent forwards. This can be useful to see exactly which traces are being forwarded to Grafana Cloud.