Troubleshoot Grafana Agent
Sometimes it can be difficult to tell what, if anything, the Grafana Agent is sending along to the backend. This document focuses on a few techniques to gain visibility on how many traces are being pushed to the Agent and if they’re making it to the backend. The tracing pipeline is built on top of the OpenTelemetry Collector which does a fantastic job of logging network and other issues.
If your logs are showing no obvious errors try the following:
The agent publishes a few Prometheus metrics that are useful to determine how much trace traffic it is receiving and successfully forwarding. These are a good place to start when diagnosing tracing Agent issues.
If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud, you can turn on Automatic Logging. A recommend debug setup is:
- name: default
This will emit logs to stdout for every root span that the Agent forwards. This can be useful to see exactly which traces are being forwarded to Grafana Cloud.