TroubleshootingMissing traces in Tempo

Some of my traces are missing in Tempo

This could happen because of a number of reasons and some have been detailed in this blog post - Where did all my spans go? A guide to diagnosing dropped spans in Jaeger distributed tracing.

Diagnosing the issue

If the pipeline is not reporting any dropped spans, check whether application spans are being dropped by Tempo. The following metrics help determine this -

  • tempo_receiver_refused_spans. The value of tempo_receiver_refused_spans should be 0. If the value of tempo_receiver_refused_spans is greater than 0, then the possible reason is the application spans are being dropped due to rate limiting.