Distributor refusing spans

Thu, 09 Apr 2026 14:59:14 +0000

Distributor refusing spans

The two most likely causes of refused spans are unhealthy ingesters or trace limits being exceeded.

To log spans that are discarded, add the --distributor.log_discarded_spans.enabled flag to the distributor or adjust the distributor configuration:

distributor:
  log_discarded_spans:
    enabled: true
    include_all_attributes: false # set to true for more verbose logs

Adding the flag logs all discarded spans, as shown below:

level=info ts=2024-08-19T16:06:25.880684385Z caller=distributor.go:767 msg=discarded spanid=c2ebe710d2e2ce7a traceid=bd63605778e3dbe935b05e6afd291006
level=info ts=2024-08-19T16:06:25.881169385Z caller=distributor.go:767 msg=discarded spanid=5352b0cb176679c8 traceid=ba41cae5089c9284e18bca08fbf10ca2

Unhealthy ingesters

Unhealthy ingesters can be caused by failing OOMs or scale down events. If you have unhealthy ingesters, your log line will look something like this:

msg="pusher failed to consume trace data" err="at least 2 live replicas required, could only find 1"

In this case, you may need to visit the ingester ring page at /ingester/ring on the Distributors and “Forget” the unhealthy ingesters. This works in the short term, but the long term fix is to stabilize your ingesters.

Trace limits reached

In high volume tracing environments, the default trace limits are sometimes not sufficient. These limits exist to protect Tempo and prevent it from OOMing, crashing, or otherwise allow tenants to not DOS each other. If you are refusing spans due to limits, you’ll see logs like this at the distributor:

msg="pusher failed to consume trace data" err="rpc error: code = FailedPrecondition desc = TRACE_TOO_LARGE: max size of trace (52428800) exceeded while adding 15632 bytes to trace a0fbd6f9ac5e2077d90a19551dd67b6f for tenant single-tenant"
msg="pusher failed to consume trace data" err="rpc error: code = FailedPrecondition desc = LIVE_TRACES_EXCEEDED: max live traces per tenant exceeded: per-user traces limit (local: 60000 global: 0 actual local: 60000) exceeded"
msg="pusher failed to consume trace data" err="rpc error: code = ResourceExhausted desc = RATE_LIMITED: ingestion rate limit (15000000 bytes) exceeded while adding 10 bytes"

You’ll also see the following metric incremented. The reason label on this metric will contain information about the refused reason.

tempo_discarded_spans_total

In this case, use available configuration options to increase limits.

Client resets connection

When the client resets the connection before the distributor can consume the trace data, you see logs like this:

msg="pusher failed to consume trace data" err="context canceled"

This issue needs to be fixed on the client side. To inspect which clients are causing the issue, logging discarded spans with include_all_attributes: true may help.

Note that there may be other reasons for a closed context as well. Identifying the reason for a closed context is not straightforward and may require additional debugging.

Troubleshoot Grafana Alloy

Thu, 09 Apr 2026 14:59:14 +0000

Troubleshoot Grafana Alloy

Sometimes it can be difficult to tell what, if anything, Grafana Alloy is sending along to the backend. This document focuses on a few techniques to gain visibility on how many trace spans push to Alloy and if they’re making it to the backend. OpenTelemetry Collector form the basis of the tracing pipeline, which does a fantastic job of logging network and other issues.

If your logs are showing no obvious errors, one of the following suggestions may help.

Metrics

Alloy publishes a few Prometheus metrics that are useful to determine how much trace traffic it receives and successfully forwards. These metrics are a good place to start when diagnosing tracing Alloy issues.

From the otelcol.receiver.otlp component:

receiver_accepted_spans_ratio_total
receiver_refused_spans_ratio_total

From the otelcol.exporter.otlp component:

exporter_sent_spans_ratio_total
exporter_send_failed_spans_ratio_total

Alloy has a Prometheus scrape endpoint, /metrics, that you can use to check metrics locally by opening a browser to http://localhost:12345/metrics. The /metrics HTTP endpoint of the Alloy HTTP server exposes the Alloy component and controller metrics. Refer to the Monitor the Grafana Alloy component controller documentation for more information.

Check metrics in Grafana Cloud

In your Grafana Cloud instance, you can check metrics using the grafanacloud-usage data source. To view the metrics, use the following steps:

From your Grafana instance, select Explore in the left menu.
Change the data source to grafanacloud-usage.
Type the metric to verify in the text box. If you start with grafanacloud_traces_, you can use autocomplete to browse the list of available metrics.

Refer to Cloud Traces usage metrics for a list of metrics related to tracing usage.

Trace span logging

If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud, you can configure Alloy to output all the traces it receives to the console.

Issues with sending traces on Grafana Labs

Distributor refusing spans

Distributor refusing spans

Unhealthy ingesters

Trace limits reached

Client resets connection

Troubleshoot Grafana Alloy

Troubleshoot Grafana Alloy

Metrics

Check metrics in Grafana Cloud

Trace span logging