Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Connect to data sourcesbreadcrumb arrow Data sourcesbreadcrumb arrow Tempobreadcrumb arrow Troubleshooting
Grafana Cloud Enterprise Open source

Troubleshoot Tempo data source issues

This document provides solutions to common issues you may encounter when configuring or using the Tempo data source in Grafana.

Scope of this guide

This guide covers issues related to connecting Grafana to Tempo and using the data source features. It applies to the following setups:

  • Self-managed Grafana + self-managed Tempo—you manage both Grafana (OSS or Enterprise) and Tempo.
  • Grafana Cloud + Cloud Traces—you use Grafana Cloud with the managed Tempo backend (Grafana Cloud Traces). Some configuration (streaming, metrics generator) is handled automatically.
  • Grafana Cloud + self-managed Tempo via PDC—you use Grafana Cloud but connect to your own Tempo instance through Private data source connect.

Where troubleshooting steps differ between these setups, the guide calls it out. Sections labeled Grafana Cloud only or self-managed Tempo apply only to those environments.

Resources for troubleshooting Tempo

For issues with Tempo itself (not the data source), refer to the Tempo product documentation:

Resources for troubleshooting tracings in Grafana Cloud

Additional resources for Grafana Cloud:

Connection errors

These errors occur when Grafana cannot establish or maintain a connection to the Tempo instance.

Failed to connect to Tempo

Error message: Failed to connect to Tempo or dial tcp: connection refused

Cause: Grafana can’t reach the Tempo instance. This typically means the Tempo process isn’t running, the URL in the data source settings is wrong, or a network-level rule is blocking the connection.

Solution:

  1. Verify that the Tempo instance is running and accessible from the Grafana server.
  2. Check that the URL is correct in the data source configuration. The default Tempo HTTP port is 3200. Common URL mistakes include:
    • Adding trailing path segments, for example, /api or /tempo, to self-managed Tempo URLs. The correct format is http://<HOST>:3200 with no path. The /tempo suffix is only used for Grafana Cloud Traces URLs.
    • Using the gRPC port (9095) instead of the HTTP port (3200). The Tempo data source URL must be the HTTP endpoint. Grafana derives the gRPC connection for streaming from the same URL.
    • Refer to the Connection section for URL format examples by environment.
  3. Ensure there are no firewall rules blocking the connection between Grafana and Tempo.
  4. For Grafana Cloud users connecting to a self-managed Tempo instance, ensure you have configured Private data source connect if Tempo isn’t publicly accessible.
  5. Verify the protocol (HTTP or HTTPS) matches your Tempo configuration.

Connection timeout

Error message: Connection timed out or context deadline exceeded

Cause: The connection to Tempo timed out before receiving a response. This can happen when network latency is high, Tempo is overloaded, or an intermediary device (load balancer, proxy) terminates the connection before Tempo responds.

Solution:

  1. Check the network latency between Grafana and Tempo.
  2. Verify that Tempo isn’t overloaded or experiencing performance issues. If you have Tempo mixin dashboards installed, check them for resource saturation or elevated error rates.
  3. Increase the Timeout setting in the data source configuration under Additional settings.
  4. Check if any network devices (load balancers, proxies) are timing out the connection.
  5. For large trace queries, consider reducing the time range or adding more specific filters.

TLS/SSL connection failures

Error message: TLS handshake failed or x509: certificate signed by unknown authority

Cause: There’s a mismatch between the TLS settings in Grafana and what the Tempo instance supports or requires. For example, Tempo may present a self-signed certificate that the Grafana server doesn’t trust, or the certificate’s hostname doesn’t match the URL.

Solution:

  1. Verify that Tempo has a valid TLS certificate if HTTPS is enabled.
  2. Check that the certificate is trusted by the Grafana server.
  3. If using a self-signed certificate, configure the TLS/SSL Auth Details in the data source settings.
  4. Ensure the certificate’s Common Name (CN) or Subject Alternative Name (SAN) matches the hostname used in the URL.

For additional information on setting up TLS encryption with Tempo, refer to Configure TLS communication.

Authentication errors

These errors occur when there are issues with authentication credentials or permissions.

Authentication failed

Error message: 401 Unauthorized or 403 Forbidden

Cause: The authentication credentials are invalid or the user doesn’t have permission to access Tempo. This error appears when you select Save & test or when running a query.

Solution:

  1. Verify that the authentication credentials (username/password or API key) are correct in the data source settings.
  2. Check that the user or service account has the required permissions.
  3. Ensure the authentication method selected in the data source matches what Tempo expects.
  4. For Grafana Cloud, verify that your Cloud Access Policy token has the traces:read scope. Refer to Create a Cloud Access Policy token for setup instructions. To find your stack URL and credentials, refer to Locate your stack URL, user, and password.
  5. For self-managed Tempo, ensure the credentials match those configured in the Tempo server block or reverse proxy.

Save and test returns 404

Error message: Tempo echo endpoint returned status 404 when selecting Save & test

Cause: The Grafana health check reaches the server but the api/echo endpoint returns a 404. The most common cause is incorrect path segments in the URL (such as /api or /tempo on a self-managed instance) that shift the health check request to a nonexistent path. A 404 can also occur when a reverse proxy in front of Tempo rejects requests that include unexpected basic authentication headers.

Solution:

  1. Verify that the URL doesn’t include trailing path segments. For self-managed Tempo, the URL should be http://<HOST>:3200 with no path. Refer to the Connection section for the correct format.
  2. If a reverse proxy sits between Grafana and Tempo, check whether the proxy is returning the 404. Tempo itself ignores unexpected auth headers, but a proxy may reject them.
  3. If Tempo doesn’t require authentication, select No authentication in the data source settings.
  4. Check that the authentication credentials are entered in the correct fields. For Grafana Cloud, the User field is the instance ID (not your Grafana login), and the Password is a Cloud Access Policy token.

Multi-tenant authentication issues

Error message: Unauthorized when using multi-tenant Tempo

Cause: The tenant ID (X-Scope-OrgID header) is missing or incorrect. This header must be set on both the write path (in your Alloy or OpenTelemetry Collector configuration) and the read path (in the Grafana data source settings) to route data to the correct tenant.

Solution:

  1. In the Tempo data source settings, verify that the X-Scope-OrgID header is configured under Authentication > HTTP Headers. Refer to Multi-tenancy for data source configuration steps.
  2. On the write path, verify that the same X-Scope-OrgID header is set in your Alloy or OpenTelemetry Collector configuration so traces are ingested under the correct tenant.
  3. Check that the authenticated user has access to the specified tenant.
  4. For cross-tenant queries, ensure all specified tenants are accessible.

For more information, refer to Enable multi-tenancy and Tenant IDs.

Query errors

These errors occur when there are issues with TraceQL queries or trace lookups.

Trace not found

Error message: Trace not found or trace ID not found

Cause: The specified trace ID doesn’t exist in Tempo or has aged out of the configured retention period. Common reasons include:

  • Sampling: Head or tail sampling in your instrumentation pipeline (Alloy, OpenTelemetry Collector) may have dropped the trace before it reached Tempo. This is the most common cause of missing traces.
  • Retention: Traces are only available for the duration set in the Tempo retention configuration. For Grafana Cloud users, default retention periods apply unless you contact Support to change them.
  • Ingestion failure: The trace may not have been successfully ingested due to rate limits, errors, or misconfigured pipelines.

Solution:

  1. Verify the trace ID is correct and complete.

  2. Check your sampling configuration in Alloy or OpenTelemetry Collector. If head or tail sampling is enabled, the trace may have been intentionally dropped. Refer to Sampling strategies for guidance.

  3. Check that the trace is within the configured retention period for Tempo.

  4. If using time range restrictions, expand the time range in the TraceID query settings.

  5. Enable Use time range in query in the TraceID query settings and adjust the time shift values to search a broader range.

  6. Verify the trace was successfully ingested by Tempo. You can run a broad query to check for recent traces:

    traceql 
    { } | count() > 0

    If this returns results, traces are being ingested but your specific trace ID may have been dropped by sampling or aged out. For more query examples, refer to the TraceQL cookbook.

  7. For Grafana Cloud users, refer to Troubleshoot TraceQL and search for TraceQL queries that can help investigate missing traces.

TraceQL syntax errors

Error message: parse error or unexpected token

Cause: The TraceQL query contains syntax errors.

Solution:

  1. Verify the query follows TraceQL syntax.
  2. Check that all braces, parentheses, and quotes are balanced.
  3. Ensure attribute names are correctly formatted with the correct scope. Use dot notation for span or resource attributes, for example, span.http.status_code or resource.service.name.
  4. Use the Search query builder to generate valid TraceQL queries if you’re unfamiliar with the syntax.
  5. Verify operators are used correctly. Supported operators include =, !=, =~, !~, >, <, >=, <=.

Query returns no results

Cause: The query is valid but doesn’t match any traces.

Solution:

  1. Expand the time range to search a broader period.
  2. Verify the attribute names and values exist in your traces.
  3. Check that the attribute names match exactly, as they’re case-sensitive.
  4. Use the Search query builder to explore available attributes and values.
  5. Start with a broader query and progressively add filters to narrow results.
  6. Try Grafana Traces Drilldown, a queryless app that lets you explore tracing data using RED metrics without writing TraceQL.
  7. Note that Tempo search is non-deterministic—identical queries can return different results because Tempo scans in parallel and returns the first matching traces. Refer to Understand search behavior for details.
  8. Verify traces are being ingested into Tempo by querying for a known trace ID.

Traces not appearing after successful connection

Cause: Save & test succeeds, but running queries in Explore returns no traces. The data source connection is valid, but no trace data is reaching Tempo, or the queries don’t match any ingested data.

Solution:

  1. Verify that your applications are instrumented and that your collector (Alloy, OpenTelemetry Collector) is forwarding spans to Tempo. Check collector logs for send errors, rejected spans, or drops.
  2. For Grafana Cloud users who send traces with Alloy, refer to Troubleshoot sending traces with Alloy for ingestion-side issues.
  3. Confirm that the query time range covers the period when traces were generated. Trace data only appears for the window in which spans were ingested.
  4. For multi-tenant setups, verify that the X-Scope-OrgID header is the same on both the write path (collector) and the read path (Grafana data source). A mismatch causes traces to be ingested under one tenant and queried under another.
  5. Try a broad query, for example, {}, with a short, recent time range to confirm that any traces exist in Tempo. If this returns results, your original query filters are too narrow.

Query timeout

Error message: context deadline exceeded or query timeout

Cause: The query took too long to execute. Trace searches scan raw trace data across the selected time range, so broad queries over long periods can exceed timeout limits.

Solution:

  1. Reduce the time range to limit the amount of data scanned.
  2. Add more specific filters to narrow the search scope.
  3. Use indexed attributes in your queries when possible for faster lookups.
  4. For self-managed Tempo, increase the query timeout in the Tempo server configuration if appropriate.
  5. If you need aggregate data (rates, counts, durations) rather than individual traces, consider using TraceQL metrics queries (for example, { } | rate()) or pre-computed metrics from the metrics generator, which are optimized for aggregate workloads.

TraceQL metrics query fails

Error message: localblocks processor not found or metrics-generator not configured

Cause: TraceQL metrics queries (such as { } | rate() or { } | count_over_time()) require the local-blocks processor to be active in the Tempo metrics-generator configuration. This processor is separate from the service-graphs and span-metrics processors and must be enabled independently.

Solution:

  1. For self-managed Tempo, activate the local-blocks processor in the overrides configuration. The configuration path depends on your Tempo version and deployment method:

    • Standard: overrides.metrics_generator_processors: ["local-blocks"]
    • Newer versions and Helm: overrides.defaults.metrics_generator.processors: [local-blocks]

    Refer to Configure TraceQL metrics for setup steps and required configuration.

  2. For Grafana Cloud, the local-blocks processor is enabled by default. If you still see this error, contact Grafana Support.

  3. Verify that the metrics generator is running and healthy by checking the Tempo metrics-generator logs for errors.

Streaming issues

Streaming displays TraceQL query results as they become available. Without streaming, you don’t see results until the query completes. Streaming is available for Grafana Cloud users by default. For self-managed Tempo, search streaming requires Tempo v2.2 or later, and metrics streaming requires Tempo v2.7 or later. Both require stream_over_http_enabled: true in the Tempo configuration.

For more information, refer to Streaming.

Streaming not working

Cause: Results don’t appear incrementally and only display after the full query completes.

Solution:

  1. For self-managed Tempo, verify that you’re using the required Tempo version (v2.2 or later for search streaming, v2.7 or later for metrics streaming) and that streaming is enabled in your configuration (stream_over_http_enabled: true). Refer to Tempo gRPC API in the Tempo documentation. Streaming is available for Grafana Cloud users by default.
  2. Verify that Streaming is enabled in your Tempo data source settings. The Search queries and Metrics queries toggles are independent—check that the toggle for the query type you’re using is enabled.
  3. If your Tempo instance is behind a load balancer or proxy that doesn’t support gRPC or HTTP2, streaming may not work. Disable streaming and use standard HTTP queries instead.
  4. If you’re using Private data source connect (PDC) to reach Tempo, verify that the PDC tunnel is running and the Tempo instance is reachable from the PDC agent. PDC connectivity issues can present as streaming timeouts.

gRPC transport security error

Error message: rpc error: code = Unavailable desc = credentials require transport level security

Cause: Grafana tries to use gRPC over an insecure (non-TLS) connection. Streaming uses gRPC, which may require TLS depending on your Tempo configuration.

Solution:

  1. Configure TLS between Grafana and Tempo. Refer to Configure TLS communication.
  2. If TLS isn’t an option, disable streaming in the Tempo data source settings and use standard HTTP queries instead.

Partial results or streaming errors

Error message: stream error or incomplete results during streaming

Cause: The streaming connection was interrupted before all results were delivered.

Solution:

  1. Check network stability between the browser and Grafana.
  2. Verify that proxies or load balancers aren’t terminating long-lived connections.
  3. Disable streaming and use standard queries if issues persist.
  4. Check Tempo server logs for any errors related to streaming.

Service graph issues

The Service Graph visualizes service dependencies and highlights request rate, error rate, and duration (RED metrics) across connections. It requires a linked Prometheus data source with service graph metrics generated by the Tempo metrics generator or Grafana Alloy. For more information, refer to Service Graph and Service Graph view.

Service graph not displaying

Cause: The Service Graph view shows no data or an empty graph.

Solution:

  1. Verify that a Prometheus data source is linked in the Tempo data source Service Graph settings under Additional settings.

  2. For self-managed Tempo, check that the metrics generator is configured to produce service graph metrics. Refer to Service graph metrics in the Tempo documentation.

  3. Ensure the following service graph metrics exist in Prometheus:

    • traces_service_graph_request_total
    • traces_service_graph_request_failed_total
    • traces_service_graph_request_server_seconds_sum

  4. Verify that Grafana Alloy or the Tempo metrics generator is configured to produce these metrics. These are distinct from span metrics (traces_spanmetrics_*), which power the Service Graph view table.

  5. To verify the metrics exist, run the following query in Explore with your Prometheus data source selected:

    promql 
    {__name__=~"traces_service_graph_request.*"}

  6. Check the Prometheus data source connection is working.

  7. For Grafana Cloud Traces users, refer to Troubleshoot service graph and RED metrics.

Service Graph view table is empty

Cause: The Service Graph node graph renders correctly, but the RED metrics table alongside it shows no data.

Solution:

  1. The Service Graph view table uses a different set of metrics from the node graph. The Grafana frontend queries traces_spanmetrics_calls_total and traces_spanmetrics_latency_bucket. Confirm that these metrics exist in your linked Prometheus data source. Newer Tempo versions may produce traces_spanmetrics_duration_seconds_bucket instead of traces_spanmetrics_latency_bucket. If you only have the newer metric name, the duration column shows no data until the Grafana frontend supports the renamed metric.
  2. Verify that span metrics generation is enabled in your Tempo or Alloy configuration. Refer to Enable service graphs in the Tempo documentation.

High cardinality warnings

Cause: Service graph or span metrics queries produce high-cardinality warnings or slow queries.

Solution:

  1. Review the label cardinality of your service graph and span metrics. High-cardinality labels (such as unique Pod names or request IDs) create many time series and degrade query performance.
  2. Refer to Cardinality in the Tempo documentation for guidance on managing label cardinality.

Service graph shows incomplete data

Cause: Some services or connections are missing from the graph.

Solution:

  1. Verify that all services are instrumented and sending spans.
  2. Check that span names and service names are consistent.
  3. Ensure the time range includes data from all expected services.
  4. Verify the Prometheus data source has metrics for all services.

Trace to logs/metrics/profiles issues

These issues relate to the correlation features that link traces to other telemetry signals. For configuration details, refer to Trace to logs, Trace to metrics, and Trace to profiles.

Cause: Links to logs don’t appear when viewing a trace.

Solution:

  1. Verify that a Loki or other log data source is configured in the Trace to logs settings.
  2. Check that the configured tags exist in your spans.
  3. Ensure at least one mapped tag has a non-empty value in the span.
  4. If using a custom query, verify the query syntax is valid for the target data source.
  5. Check that Filter by trace ID or Filter by span ID aren’t enabled when using a custom query.
  6. If using Filter by trace ID with Loki derived fields, verify that the derived field regular expression matches the trace ID field name used in your logs. Different instrumentation libraries use different field names (traceID, trace_id, or traceId). Refer to Trace to logs for common regular expression patterns by instrumentation format.

Cause: The reverse direction—navigating from a Loki log line to a trace—isn’t working. No trace link appears on log lines.

Solution:

  1. Verify that a derived field is configured in the Loki data source under Additional settings > Derived fields. The derived field extracts the trace ID from log lines and creates a link to Tempo.
  2. Check that the derived field regular expression matches the trace ID format in your logs. Use Show example log message in the derived field settings to test the regular expression against a sample log line.
  3. Confirm the Internal link toggle is enabled and points to the correct Tempo data source.
  4. If different services use different trace ID field names, you need a separate derived field entry for each format. Refer to Trace to logs for regular expression patterns by format.

Cause: Links to metrics don’t appear when viewing a trace.

Solution:

  1. Verify that a Prometheus or other metrics data source is configured in the Trace to metrics settings.
  2. Check that the configured tags exist in your spans.
  3. If using custom queries, verify they’re valid PromQL.
  4. Ensure the $__tags variable is correctly placed in custom queries.

Cause: Links to profiling data don’t appear when viewing a trace.

Solution:

  1. Verify that a Grafana Pyroscope data source is configured in the Trace to profiles settings.
  2. Check that the configured tags exist in your spans.
  3. Verify that the configured profile type matches what Pyroscope has data for (for example, process_cpu:cpu:nanoseconds:cpu:nanoseconds). A mismatch between the configured profile type and the available data causes the link to not appear.
  4. Tighten the Span start time shift and Span end time shift values. Profiles are sampled at intervals, so a narrow time window around the span is more likely to match profile data than a wide one.
  5. Verify that the profiling data source is connected and returning data.

Cause: The generated query doesn’t match any data in the target data source.

Solution:

  1. Adjust the Span start time shift and Span end time shift to widen the time range.
  2. Verify that log or metric labels match the span attributes.
  3. Check that the tag mappings correctly translate attribute names between data sources. Span attributes use dots (for example, service.name) but Loki labels use underscores (for example, service_name). Ensure tag mappings account for this difference. Refer to Trace to logs for tag mapping configuration.
  4. Run the generated query directly in the target data source’s Explore view to confirm it returns data outside of the trace context.
  5. Use the Query Inspector to view the generated query and verify it’s correct.

Performance issues

These issues relate to slow queries or high resource usage.

Slow trace queries

Cause: Queries take a long time to return results.

Solution:

  1. Reduce the dashboard or Explore time range.
  2. Add more specific filters to narrow the search scope.
  3. Use indexed attributes in TraceQL queries when possible.
  4. Enable streaming to see results as they become available.
  5. Consider using the Search query builder which can help optimize queries.

High memory usage during queries

Cause: Large traces or result sets consume excessive memory.

Solution:

  1. Reduce the Limit setting in query options to return fewer traces.
  2. Reduce the Span Limit setting to return fewer spans per trace.
  3. Add filters to reduce the result set size.
  4. Avoid querying very large time ranges.

Other common issues

The following issues don’t produce specific error messages but are commonly encountered.

Node graph not displaying

Cause: The node graph visualization doesn’t appear above the trace view.

Solution:

  1. Verify that Node graph is enabled in the Tempo data source settings under Additional settings.
  2. Check that the trace contains multiple spans with parent-child relationships. Spans must include traceID and spanID, and non-root spans must include parentSpanID to establish hierarchy. If these identifiers are missing, the trace data is incomplete at the instrumentation level and the graph can’t be rendered.

Autocomplete not working

Cause: The TraceQL editor doesn’t show suggestions for attributes or values. Autocomplete requires Tempo to enumerate tags, which requires a supported storage backend (vParquet or later).

Solution:

  1. For self-managed Tempo, verify that the storage backend supports tag search. Tempo requires vParquet format or later for tag-based search and autocomplete. Refer to Parquet configuration for details. Grafana Cloud Traces supports autocomplete by default.
  2. Check the Tags time range setting in the data source settings to ensure it covers a period with recent trace data.
  3. Increase the Tag limit setting if you have many unique attributes.
  4. Verify the Tempo data source connection is working by selecting Save & test in the data source settings.

Data source settings disappear after restart

Cause: You configured settings in the Grafana UI (such as trace-to-logs links, service graph, or span time shifts), but they disappear after Grafana restarts. This happens when the data source is provisioned via a YAML configuration file, Helm chart, or Terraform. Provisioned data sources are read-only in the UI—any changes you make through the UI aren’t persisted and silently revert on the next restart.

You can tell a data source is provisioned if the settings form is read-only and the button reads Test instead of Save & test.

Solution:

  1. To change settings permanently, edit the provisioning YAML file and restart Grafana (or wait for the provisioning system to reload). Any setting available in the UI can be set in the YAML file, including trace-to-logs, trace-to-metrics, and service graph configuration. Refer to Provision the Tempo data source for the full YAML reference.
  2. For Grafana Cloud, you can clone the provisioned data source to create an editable copy that persists UI changes. Refer to Clone a provisioned data source for Grafana Cloud for the steps.

TraceQL alerting not available

Cause: You can select the Tempo data source in the Grafana Alerting rule builder, but standard TraceQL search queries return trace data (spans and trace IDs), not numeric time series. Alert rules require numeric data to evaluate thresholds, so trace search queries don’t produce usable alert conditions.

Solution:

TraceQL metrics queries (such as { } | rate() or { status = error } | count_over_time()) return numeric time series and can drive alert thresholds directly. However, TraceQL alerting requires the experimental tempoAlerting feature flag, which is disabled by default. To enable it, set the tempoAlerting feature flag in your Grafana configuration. For Grafana Cloud, contact Support to enable it. For step-by-step setup, refer to Trace-based alerts.

Caution

The tempoAlerting feature flag is experimental. For production alerting, consider the Prometheus-based approach described below.

Alternatively, use the Tempo metrics generator to produce Prometheus metrics from trace data, then create alert rules against the Prometheus data source that stores those metrics.

  1. Configure the Tempo metrics generator (using the service-graphs, span-metrics, or local-blocks processor) to produce metrics from traces and remote-write them to a Prometheus-compatible backend.
  2. Create alert rules against the Prometheus data source that receives those metrics. For example, alert on high error rates using traces_spanmetrics_calls_total with a status_code="STATUS_CODE_ERROR" filter.
  3. Refer to the feature support table in the Tempo data source overview for details on alerting support.

Switching between multiple Tempo instances

Cause: You have multiple Tempo data sources configured (for example, one per environment or per tenant), but there’s no built-in UI dropdown in Explore or dashboards to switch between them dynamically.

Workaround:

  1. In dashboards, create a data source template variable filtered to tempo type. This adds a dropdown that lets viewers switch between Tempo instances without editing the dashboard.
  2. In Explore, use the data source picker in the top bar to switch between configured Tempo data sources.
  3. For multi-tenant setups on a single Tempo instance, consider using the X-Scope-OrgID header instead of separate data sources. Refer to Multi-tenancy for configuration details.

Private data source connect issues (Grafana Cloud only)

These issues relate to connecting to a private Tempo instance using Private data source connect (PDC). PDC is only available in Grafana Cloud.

Note

Private data source connect allows Grafana Cloud to securely connect to data sources in your private network without exposing them to the public internet. For setup instructions, refer to Private data source connect.

PDC agent not connected

Error message: No healthy PDC agents available or Private data source connect agent not connected

Cause: The PDC agent installed in your private network isn’t running or can’t connect to Grafana Cloud.

Solution:

  1. Verify the PDC agent is running in your private network.
  2. Check the PDC agent logs for connection errors.
  3. Ensure the PDC agent has outbound network access to Grafana Cloud endpoints.
  4. Verify the PDC agent is using the correct token for your Grafana Cloud instance.
  5. Check that firewall rules allow outbound HTTPS (port 443) connections from the PDC agent.
  6. Restart the PDC agent if it appears stuck or unresponsive.

For more information, refer to Private data source connect.

Connection through PDC failing

Error message: Failed to connect through private data source or Connection refused via PDC

Cause: The PDC agent is connected to Grafana Cloud but cannot reach the Tempo instance in your private network.

Solution:

  1. Verify the Tempo URL is correct and accessible from the machine running the PDC agent.
  2. Check that the PDC agent can resolve the Tempo hostname (DNS resolution).
  3. Ensure there are no firewall rules blocking connections from the PDC agent to Tempo.
  4. Verify the Tempo port (default 3200) is open and accessible.
  5. Test connectivity from the PDC agent host to Tempo using curl or telnet.

PDC timeout errors

Error message: Request timed out via private data source connect

Cause: Queries through the PDC tunnel are timing out.

Solution:

  1. Check network latency between the PDC agent and Tempo.
  2. Verify the PDC agent host has sufficient resources (CPU, memory, network bandwidth).
  3. Ensure no network devices between the PDC agent and Tempo are dropping long-running connections.
  4. For large queries, consider reducing the time range or adding more specific filters.
  5. Check if multiple data sources are sharing the same PDC agent, which may cause resource contention.

PDC authentication mismatch

Error message: Unauthorized or 403 Forbidden when using PDC

Cause: The authentication configured in the data source doesn’t work with the PDC connection.

Solution:

  1. Verify that authentication credentials are configured correctly in the Grafana Cloud data source settings.
  2. Check that the Tempo instance accepts the authentication method configured.
  3. Ensure any required headers (such as X-Scope-OrgID for multi-tenant Tempo) are properly forwarded through PDC.
  4. Test the connection directly from the PDC agent host to verify credentials work outside of PDC.

Get additional help

If you continue to experience issues after following this troubleshooting guide:

  1. For Grafana Cloud users, try Grafana Assistant to investigate trace issues interactively (if enabled on your account).
  2. For Grafana Cloud users who send traces with Alloy, refer to the Alloy troubleshooting steps for ingestion-side issues.
  3. Ask questions in the Grafana community forums or the Grafana Community Slack.
  4. Review the Grafana GitHub issues for known bugs.
  5. Review the Tempo GitHub issues for Tempo-specific bugs.
  6. Enable debug logging in Grafana to capture detailed error information.
  7. Check Tempo server logs for additional error details.
  8. Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Contracted user.

When reporting issues, include:

  • Grafana version
  • Tempo version
  • Error messages (redact sensitive information)
  • Steps to reproduce
  • TraceQL query examples (redact sensitive data)
  • Relevant configuration settings