Tempo data source on Grafana Labs

Configure the Tempo data source

Tue, 14 Apr 2026 14:59:56 +0000

Configure the Tempo data source

The Tempo data source connects Grafana to your Tempo database and lets you configure features and integrations with other telemetry signals.

You can configure the data source using either the data source interface in Grafana or using a configuration file. This page explains how to connect Grafana to Tempo, set up authentication, and enable streaming. For trace correlation features, provisioning, and other settings, refer to the linked sub-pages.

Depending upon your tracing environment, you may have more than one Tempo instance. Grafana supports multiple Tempo data sources.

Before you begin

To configure a Tempo data source, you need:

Administrator rights to your Grafana instance.
A Tempo instance configured to send tracing data to Grafana.
An application or service instrumented to emit traces. Refer to Instrument for tracing if you haven’t set this up yet.

If you’re provisioning a Tempo data source, you also need administrative rights on the server hosting your Grafana instance. Refer to Provision the Tempo data source for provisioning instructions.

Note
You can’t modify a provisioned data source from the Grafana UI. The settings form is read-only and the Save & test button is replaced by Test (you can test the connection but not save changes).

To make changes, you can either:

Clone the data source: Create a new data source of the same type and copy the settings from the provisioned data source. Refer to Clone a provisioned data source for Grafana Cloud.

Update the provisioning file: Edit the YAML configuration file. Refer to Provision the Tempo data source.

Add or modify a data source

You can use these procedures to configure a new Tempo data source or to edit an existing one.

Add a new data source

Follow these steps to set up a new Tempo data source:

Select Connections in the main menu.
Enter Tempo in the search bar.
Select Tempo.
Select Add new data source in the top-right corner of the page.
On the Settings tab, complete the Name, Connection, and Authentication sections.
- Use the Name field to specify the name used for the data source in panels, queries, and Explore. Toggle the Default switch for the data source to be pre-selected for new panels.
- Under Connection, enter the URL of the Tempo instance. Refer to the connection section for URL format examples.
- Complete the Authentication section.
Optional: Configure other sections to add capabilities to your tracing data. Refer to the Configure trace correlations and Other settings sections for available options.
Select Save & test.

This video explains how to add data sources, including Loki, Tempo, and Mimir, to Grafana and Grafana Cloud. Tempo data source setup starts at 4:58 in the video.

Update an existing data source

To modify an existing Tempo data source:

Select Connections in the main menu.
Select Data sources to view a list of configured data sources.
Select the Tempo data source you want to modify.
Configure or update additional sections to add capabilities to your tracing data.
After completing your updates, select Save & test.

Connection

The URL format depends on your environment:

Environment	URL format	Example
Self-managed Tempo	`http://<TEMPO_HOST>:<PORT>`	`http://tempo:3200`
Grafana Cloud Traces	`https://tempo-<REGION>.grafana.net/tempo`	`https://tempo-prod-01-eu-west-0.grafana.net/tempo`

The default port depends on the protocol: 3200 for HTTP, 9095 for gRPC.

Note
Grafana Cloud includes Grafana Cloud Traces, a pre-configured tracing data source backed by Tempo. Add a Tempo data source manually when you need to connect to a self-managed Tempo instance or require custom configuration beyond what Grafana Cloud Traces provides.

The URLs in this table are for querying traces from Grafana. To configure how your applications send traces to Tempo (OTLP endpoints), refer to Set up tracing for Grafana Cloud or Instrument for tracing for self-managed Tempo.

Authentication

Use this section to select an authentication method to access the data source.

Note
Use Transport Layer Security (TLS) for an additional layer of security when working with Tempo. For additional information on setting up TLS encryption with Tempo, refer to Configure TLS communication and Tempo configuration.

To set up authentication:

Select an authentication method from the drop-down list:
- Basic authentication: Authenticates your data source using a username and password
- Forward OAuth identity: Forwards the OAuth access token and the OIDC ID token, if available, of the user querying to the data source
- No authentication: No authentication is required to access the data source
For Basic authentication only: Enter the User and Password.
Optional: Complete the TLS settings for additional security methods.

TLS Client Authentication

Toggle on to use client authentication. When enabled, it adds the Server name, Client cert, and Client key fields. The client provides a certificate that is validated by the server to establish the client’s trusted identity. The client key encrypts the data between client and server. These details are encrypted and stored in the Grafana database.

Add self-signed certificate

Activate this option to use a self-signed TLS certificate. You can add your own Certificate Authority (CA) certificate on top of one generated by the certificate authorities for additional security measure.

Skip TLS certification validation

When activated, it bypasses TLS certificate verification. Not recommended, unless absolutely necessary for testing.
Optional: Add HTTP Headers. You can pass along additional context and metadata data about the request and response. Select Add header to add Header and Value fields.
Select Save & test to preserve your changes.

Connecting to Grafana Cloud Traces: Select Basic authentication. Use your Grafana Cloud stack’s instance ID (shown in the Tempo data source URL on the Cloud Portal) as the User, and a Cloud Access Policy token with traces:read scope as the Password. Don’t use your Grafana login credentials.

Self-managed Tempo: If Tempo is configured without authentication, select No authentication. If your Tempo instance uses basic auth, enter the credentials configured in your Tempo server block or reverse proxy.

Multi-tenancy

If you’re using a multi-tenant Tempo deployment, you need to send a tenant ID with each request so Tempo routes queries to the correct tenant.

To configure multi-tenancy:

In the Tempo data source settings, scroll to the Authentication section.
Expand the HTTP Headers subsection.
Add a header with the name X-Scope-OrgID and the value set to your tenant ID.

This header is set in the Grafana data source for querying. The same header must also be set on the write path (in your Alloy or OpenTelemetry Collector configuration) to route trace data to the correct tenant on ingest.

For more information, refer to the Tempo multi-tenancy documentation.

Streaming

Streaming enables TraceQL query results to be displayed as they become available. Without streaming, no results are displayed until all results have returned.

To use streaming, you need to:

Run Tempo version 2.2 or later, or Grafana Enterprise Traces (GET) version 2.2 or later, or use Grafana Cloud Traces.
Grafana Cloud Traces: Streaming is supported out of the box. No additional infrastructure configuration is required.
Self-managed Tempo or GET: Tempo must have stream_over_http_enabled: true in its configuration for streaming to work. If your Tempo or GET instance is behind a load balancer or proxy that doesn’t support gRPC or HTTP2, streaming may not work and should be deactivated.

For more information, refer to Tempo gRPC API.

Activate streaming

Streaming is available in Grafana and Grafana Cloud. The Streaming section has two toggles:

Search queries: Streams TraceQL search results as they arrive.
Metrics queries: Streams TraceQL metrics query results as they arrive.

Enable one or both depending on which query types you use.

When streaming is active, it shows as Enabled in Explore. To check the status, select Explore in the menu, select your Tempo data source, and expand the Options section.

Troubleshoot streaming

If streaming isn’t working, refer to Streaming issues in the troubleshooting guide.

Connect traces to other signals

Grafana provides several ways to link traces to other telemetry signals. The following diagram shows the available correlation mechanisms, the direction each goes, and where each is configured.

The Tempo data source settings page includes three sections for linking from spans to other signals. Each corresponds to a section in the settings form:

Trace to logs: Navigate from spans to related logs in Loki or another log data source.
Trace to metrics: Link spans to metrics queries in Prometheus or other metrics data sources.
Trace to profiles: Link spans to profiling data in Grafana Pyroscope with embedded flame graphs.

For more flexible, rule-based correlations that can target any data source or external URL, use Grafana Trace correlations. Trace correlations are configured under Configuration > Correlations, not in the Tempo data source settings.

To link from logs or metrics to traces (the reverse direction), refer to:

Derived fields in the Loki data source (for logs to traces). The trace to logs page also covers this setup.
Exemplars in the Prometheus data source (for metrics to traces).

Other settings

Additional settings: Configure Service graph, node graph, search, TraceID query, span bar, and other settings.
Provision the Tempo data source: Configure the Tempo data source using a YAML file and clone provisioned data sources.

Custom query variables

To use a variable in your trace to logs, metrics, or profiles configuration, you need to wrap it in ${}. For example, ${__span.name}.

Variable name	Description
__tags	This variable uses the tag mapping from the UI to create a label matcher string in the specific data source syntax. The variable only uses tags that are present in the span. The link is still created even if only one of those tags is present in the span. You can use this if all tags are not required for the query to be useful.
__span.spanId	The ID of the span.
__span.traceId	The ID of the trace.
__span.duration	The duration of the span.
__span.name	Name of the span.
__span.tags	Namespace for the tags in the span. To access a specific tag named `version`, you would use `${__span.tags.version}`. In case the tag contains dot, you have to access it as `${__span.tags["http.status"]}`.
__trace.traceId	The ID of the trace.
__trace.duration	The duration of the trace.
__trace.name	The name of the trace.

Query tracing data

Tue, 14 Apr 2026 14:59:56 +0000

Query tracing data

The Tempo data source’s query editor helps you query and display traces from Tempo in Explore. The queries use TraceQL, the query language designed specifically for tracing.

For general documentation on querying data sources in Grafana, refer to Query and transform data.

Tip
Don’t know TraceQL? Try Grafana Traces Drilldown, an intuitive, queryless app that lets you explore your tracing data using RED metrics.

Before you begin

You can compose TraceQL queries in Grafana and Grafana Cloud using Explore and a Tempo data source.

Note
Before running queries, verify that your Tempo data source is configured and connected. If queries return no results or errors, refer to Configure the Tempo data source to check your connection and authentication settings.

You don’t have to know TraceQL to create a query. The Search query builder lets you select options from drop-down lists to build a query visually. These selections generate a TraceQL query that you can transfer to the TraceQL editor to refine further.

TraceQL queries follow the pattern { conditions } | pipeline, where conditions inside {} select spans and pipeline operators like count() or avg() aggregate results. To learn the full syntax, refer to Construct a TraceQL query. For a language overview, refer to the TraceQL documentation.

Choose a query editing mode

The query editor has three Query types that you can use to explore your tracing data. You can use these modes by themselves or in combination to create building blocks to generate custom queries. Adding another query adds a new query block. Refer to Use query types together for more information.

Search query builder

Start here if you’re exploring data or learning TraceQL. The Search query builder provides drop-down lists and text fields to build a query visually without needing to know TraceQL syntax. Your selections automatically generate a TraceQL query that you can copy to the editor for further refinement.

Refer to Search traces using the query builder for more information.

TraceQL query editor

Use the TraceQL editor when you need complex filters, structural queries across parent and child spans, or aggregations that the Search builder doesn’t support. The editor provides autocomplete for attribute names, scopes, and operators. You can also search for a trace ID by entering it directly into the query field.

For copy-paste query examples, refer to TraceQL query examples. Refer to Write TraceQL queries with the editor for more information.

Service graph view

The Service Graph view answers a different question than trace search: instead of finding individual traces, it visualizes how your services connect and highlights request rate, error rate, and duration (RED metrics) across those connections.

Using the Service Graph view, you can:

Discover spans that consistently return errors and the rates at which they occur.
Get an overview of the overall rate of span calls throughout your services.
Determine how long the slowest queries in your service take to complete.
Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals).

The Service Graph requires metrics generation to be configured. For more information, refer to Service Graph and Service Graph view.

Understand search behavior

Tempo search is non-deterministic. When you run a search, Tempo performs a massively parallel scan over the selected time range and returns the first N matching results. Even identical searches can return different results due to variations in machine load and network latency.

This design prioritizes speed over predictability. To make results deterministic, append a with clause to your TraceQL queries, for example { status = error } with (most_recent=true), so Tempo returns the most recent matching traces. For more information, refer to Retrieve most recent results.

Use TraceQL panels in dashboards

To add TraceQL panels to your dashboard, refer to the Traces panel documentation.

To learn more about Grafana dashboards, refer to the Use dashboards documentation.

For example dashboards with traces, visit Grafana Play.

Set options for query builder and editor

The following options are available for the Search and TraceQL query types. You can modify these settings in the Options section.

After changing any option, re-run the query to apply the updates.

Limit: Determines the maximum number of traces to return. Default value is 20.
Span Limit: Sets the maximum number of spans to return for each spanset. Default value is 3.
Table Format: Determines whether the query results table is displayed focused on Traces or Spans. Traces is the default selection. When Traces is selected, the results table starts with the trace ID. When Spans is selected, the table starts with the trace service.
Step: Defines the step for metrics queries. Use duration notation, for example, 30ms or 1m.
Streaming: Indicates if streaming is active. Streaming lets you view partial query results before the entire query completes. Activating streaming adds the Table - Streaming Progress section to the query results.

Use query types together

You can use + Add query to create customized queries that use one or more of the query types together. Each time you add a new query, it adds a new section, or query block, that contains Search, TraceQL, or Service Graph user interface.

The added query and results table appear in the navigation under Queries and Tables respectively. You can use the navigation to view query, results table, and service graph blocks.

Navigating through the query blocks

To add a query block:

Select + Add query.
Choose a query type: Search, TraceQL, or Service Graph.

To remove a query block, select the Remove query trash can icon.

To rename a block, select the Rename edit icon next to the query block name. The name changes in the queries and table list.

Additional query block options

Each query block has a set of icons in the right top corner.

These icons include:

Show data source help: Displays the Tempo Cheat Sheet with links to documentation.
Create recorded query: Lets you save the current query block as a recorded query. This option is available in Grafana Cloud and Grafana Enterprise. For more information, refer to Recorded queries.
Duplicate query: Copies the current block and adds a new identical block.
Remove query: Deletes the query block.

Use query history and query inspector

Explore provides a history of all queries you’ve used within a data source and an inspector that lets you view stats, inspect queries, view JSON, and general information for your data source queries.

For more information, refer to the Query inspector in Explore and Query management in Explore documentation.

Upload a JSON trace file

You can upload a JSON file that contains a single trace and visualize it. If the file has multiple traces, Grafana visualizes the first trace.

To upload a trace file:

Select Explore in Grafana.
Select Import trace next to the query type selector.
Upload your JSON trace file.

To download a trace or Service Graph through the Inspector panel:

Open the inspector.
Navigate to the Data tab.
Click Download traces or Download Service Graph.

Cross-tenant TraceQL queries

If you’ve configured a multi-stack Tempo data source, you can perform TraceQL queries across those stacks and tenants.

Queries performed using the cross-tenant configured data source, in either Explore or inside of dashboards, are performed across all the tenants that you specified in the X-Scope-OrgID header.

TraceQL queries that compare multiple spansets may not correctly return all traces in a cross-tenant query. For instance,

{ span.attr1 = "bar" } && { span.attr2 = "foo" }

TraceQL evaluates a contiguously stored trace. If these two conditions are satisfied in separate tenants, then Tempo doesn’t return the trace.

Refer to Set up a multi-stack Tempo data source in Grafana for information about configuring the Tempo data source.

For information about Tempo configuration requirements, refer to the Cross-tenant query and Enable multitenancy documentation.

Next steps

TraceQL query examples - Copy-paste query examples for common use cases
Construct a TraceQL query - Full TraceQL syntax, scopes, and operators
Service Graph and Service Graph view - Visualize service dependencies and RED metrics
Span filters - Refine trace results after querying
Configure the Tempo data source - Connection, authentication, and feature settings

Service Graph and Service Graph view

Tue, 14 Apr 2026 14:59:56 +0000

Service Graph and Service Graph view

The Service Graph is a visual representation of the relationships between services in a distributed system. Each node represents a service such as an API or database, and each edge represents requests between services.

Use the Service Graph to detect performance issues, track error and latency trends across services, and navigate to the traces that caused them.

The Service Graph is built on RED metrics, a monitoring methodology that focuses on three signals:

Rate: requests per second
Errors: failed requests per second
Duration: latency distribution (for example, p50, p95, p99)

Service graph data flows from traces through a metrics pipeline into Grafana:

Your application emits traces to Tempo.
The Tempo metrics generator or Grafana Alloy processes the traces and writes service graph metrics to a Prometheus-compatible backend.
Grafana reads those metrics from your linked Prometheus data source and renders the graph.

Before you begin

To use the Service Graph, you need:

Service graph metrics generation configured in Grafana Alloy or the Tempo metrics-generator. Without this, no service graph data exists.
A Prometheus data source in Grafana that receives the generated metrics.
The Prometheus data source linked in the Tempo data source settings under Service graph. Refer to Additional settings for details, or Provision the data source for YAML configuration.

If you run a Service Graph query and see an empty result, refer to Troubleshoot on this page.

How it works

The metrics generator or Alloy inspects trace spans for parent-child relationships that represent a request, using OpenTelemetry semantic conventions. It writes the following metrics to your Prometheus backend:

traces_service_graph_request_total: Total requests between two services (counter).
traces_service_graph_request_failed_total: Failed requests between two services (counter).
traces_service_graph_request_server_seconds: Response time as observed by the server (histogram).
traces_service_graph_request_client_seconds: Response time as observed by the client (histogram).

Grafana reads these metrics from the linked Prometheus data source to render the node graph and calculate request rates, error rates, and response times.

For the full metrics reference, including labels, virtual nodes, and additional metrics, refer to Service graphs in the Tempo documentation.

Display the Service Graph

To display the Service Graph:

Navigate to Explore.
Select the Tempo data source.
Select the Service Graph query type.
Run a query.
(Optional) Filter by service name.

Each circle in the graph represents a service. Numbers inside the circles indicate the response time and requests per second.

Each circle’s color shows the proportion of successful and failed requests:

Color	Meaning
Green	Success
Red	Fault
Yellow	Errors
Purple	Throttled responses

The color mix of each circle shows the proportion of each state. A mostly green circle indicates healthy traffic; visible red, yellow, or purple arcs indicate faults, errors, or throttling respectively.

To open a context menu with links for quick navigation to traces, metrics, and related views, click a service.

For details on navigating, zooming, and interacting with the graph layout, refer to the Node graph panel documentation.

To investigate a service with a high error rate, use TraceQL:

{ resource.service.name = "<SERVICE_NAME>" && status = error }

Replace <SERVICE_NAME> with the name shown in the Service Graph node. Refer to TraceQL query examples for more queries you can run from Service Graph observations.

Service Graph view

The Service Graph view displays a table of RED metrics (rate, error rate, duration) alongside the node graph. The table uses a different set of metrics from the node graph:

traces_spanmetrics_calls_total: Rate and error rate columns.
traces_spanmetrics_duration_seconds_bucket: Duration column.

These span metrics must be present in your linked Prometheus data source. Span metrics generation must be enabled in your Tempo or Alloy configuration. Refer to Enable service graphs in the Tempo documentation.

For details on the Service Graph view, refer to the Service Graph view documentation.

To open the Service Graph view, follow the same steps as Display the Service Graph. The table appears alongside the node graph after you run the query.

To open a query in Prometheus with the span name of that row automatically set in the query, click a row in the rate, error rate, or duration columns.

To open a query in Tempo with the span name of that row automatically set in the query, click a row in the links column.

You can also use TraceQL to query the same data programmatically. Refer to TraceQL query examples to translate Service Graph observations into queries.

Troubleshoot

If the Service Graph isn’t displaying data, the table is empty, or you see high cardinality warnings, refer to Service graph issues in the troubleshooting guide.

Next steps

Enable service graphs - Set up metric generation in Tempo or Alloy.
Node graph panel - Navigate and customize the graph layout.

Span filters

Tue, 14 Apr 2026 14:59:56 +0000

Span filters

When working with traces, especially those comprising a vast number of spans, pinpointing specific spans of interest can be a daunting task. This is where span filtering comes in. Located above the trace timeline viewer, the Filters bar lets you refine the spans displayed based on specific criteria. Whether you’re looking to identify spans from a certain service, those exceeding a particular duration, or spans tagged with specific attributes, span filtering streamlines the process.

The more filters you add, the more specific the results become.

Tip
If you arrived from the Service Graph, use service.name and status = error filters to drill into the specific service nodes showing errors.

Filter types

The Filters text box lets you pick a key, choose an operator, and enter or select a value.

Available filter keys include:

Text search: free-text search across span content
duration: filter by span duration (operators: =, >=, <=, >, <)
service.name: the service that produced the span
span.name: the operation name of the span
status: span status (ok, error, unset)
kind: span kind (for example, server, client, internal)
Any span tag, process tag, or log field key present in the trace

For most attribute keys, the available operators are =, !=, =~ (regular expression match), and !~ (regular expression not match).

Note
The same attributes available as filter keys (for example, span.name, service.name, duration) can be used directly in TraceQL queries.

Quick filter pills

Below the text box, three quick-filter pills let you apply common filters with a single click:

Pill	Effect
Critical path	Highlights spans in the critical path. The longest sequence of dependent tasks that determines the trace’s minimum duration.
Errors	Selects spans where `status = error`.
High latency	Selects the 10% longest spans in the trace (p90 threshold).

Click a pill to toggle the filter on or off.

Use span filters

The Filters bar is always visible above the trace timeline viewer. To add a filter:

Click the Filters text box (placeholder text: Filter by attribute or text).
Select a key from the dropdown (for example, service.name or duration).
Choose an operator (for example, = or >=).
Select or type a value. The text box suggests values extracted from the current trace.

The trace timeline viewer updates as you add filters, highlighting the matching spans.

Show all spans

By default, the Show all spans toggle is enabled, meaning all spans remain visible with matching spans highlighted. Turn off Show all spans to hide non-matching spans and display only the spans that meet your filter criteria. This is particularly useful when sifting through thousands of spans, allowing you to zero in on those that truly matter.

The toggle is disabled when no filters are active.

Navigate between matches

Use the up and down arrow buttons next to the match count to jump between matching spans in the trace timeline. The match count displays the current position and total matches (for example, 2/5 matches).

Remove a filter

To remove a filter, click the × on the filter chip below the Filters text box.

Common use cases

To find all errors from a specific service, select service.name = checkout, then click the Errors pill.

To isolate slow database calls, select span.name =~ db\..*, then select duration > 200ms.

To narrow to the critical path only, click the Critical path pill, then turn off Show all spans.

Next steps

Service Graph: Visualize service relationships and identify error-producing nodes.
Query editor: Build TraceQL queries using the same attributes available as span filter keys.
Configure the Tempo data source: Set up your Tempo data source and trace correlations.

Troubleshoot Tempo data source issues

Thu, 16 Apr 2026 20:21:47 +0000

Troubleshoot Tempo data source issues

This document provides solutions to common issues you may encounter when configuring or using the Tempo data source in Grafana.

Scope of this guide

This guide covers issues related to connecting Grafana to Tempo and using the data source features. It applies to the following setups:

Self-managed Grafana + self-managed Tempo—you manage both Grafana (OSS or Enterprise) and Tempo.
Grafana Cloud + Cloud Traces—you use Grafana Cloud with the managed Tempo backend (Grafana Cloud Traces). Some configuration (streaming, metrics generator) is handled automatically.
Grafana Cloud + self-managed Tempo via PDC—you use Grafana Cloud but connect to your own Tempo instance through Private data source connect.

Where troubleshooting steps differ between these setups, the guide calls it out. Sections labeled Grafana Cloud only or self-managed Tempo apply only to those environments.

Resources for troubleshooting Tempo

For issues with Tempo itself (not the data source), refer to the Tempo product documentation:

Troubleshoot Tempo - General Tempo troubleshooting, including ingestion failures and server-side errors.
Unable to find traces - Traces missing due to ingestion, sampling, or retention issues.
Too many requests error - Query capacity limits and 429 errors.
Query issues - Server-side query failures, bad blocks, and performance tuning.

Resources for troubleshooting tracings in Grafana Cloud

Additional resources for Grafana Cloud:

Troubleshoot Grafana Cloud Traces, which covers quick checks, ingestion issues, TraceQL and search, service graph, exemplars, and rate limiting and retry.
Investigate traces with Grafana Assistant - Use Grafana Assistant to help troubleshoot any issues.
Troubleshoot traces collection with Alloy
Troubleshoot errors with metrics-generator in Cloud Traces

Connection errors

These errors occur when Grafana cannot establish or maintain a connection to the Tempo instance.

Failed to connect to Tempo

Error message: Failed to connect to Tempo or dial tcp: connection refused

Cause: Grafana can’t reach the Tempo instance. This typically means the Tempo process isn’t running, the URL in the data source settings is wrong, or a network-level rule is blocking the connection.

Solution:

Verify that the Tempo instance is running and accessible from the Grafana server.
Check that the URL is correct in the data source configuration. The default Tempo HTTP port is 3200. Common URL mistakes include:
- Adding trailing path segments, for example, /api or /tempo, to self-managed Tempo URLs. The correct format is http://<HOST>:3200 with no path. The /tempo suffix is only used for Grafana Cloud Traces URLs.
- Using the gRPC port (9095) instead of the HTTP port (3200). The Tempo data source URL must be the HTTP endpoint. Grafana derives the gRPC connection for streaming from the same URL.
- Refer to the Connection section for URL format examples by environment.
Ensure there are no firewall rules blocking the connection between Grafana and Tempo.
For Grafana Cloud users connecting to a self-managed Tempo instance, ensure you have configured Private data source connect if Tempo isn’t publicly accessible.
Verify the protocol (HTTP or HTTPS) matches your Tempo configuration.

Connection timeout

Error message: Connection timed out or context deadline exceeded

Cause: The connection to Tempo timed out before receiving a response. This can happen when network latency is high, Tempo is overloaded, or an intermediary device (load balancer, proxy) terminates the connection before Tempo responds.

Solution:

Check the network latency between Grafana and Tempo.
Verify that Tempo isn’t overloaded or experiencing performance issues. If you have Tempo mixin dashboards installed, check them for resource saturation or elevated error rates.
Increase the Timeout setting in the data source configuration under Additional settings.
Check if any network devices (load balancers, proxies) are timing out the connection.
For large trace queries, consider reducing the time range or adding more specific filters.

TLS/SSL connection failures

Error message: TLS handshake failed or x509: certificate signed by unknown authority

Cause: There’s a mismatch between the TLS settings in Grafana and what the Tempo instance supports or requires. For example, Tempo may present a self-signed certificate that the Grafana server doesn’t trust, or the certificate’s hostname doesn’t match the URL.

Solution:

Verify that Tempo has a valid TLS certificate if HTTPS is enabled.
Check that the certificate is trusted by the Grafana server.
If using a self-signed certificate, configure the TLS/SSL Auth Details in the data source settings.
Ensure the certificate’s Common Name (CN) or Subject Alternative Name (SAN) matches the hostname used in the URL.

For additional information on setting up TLS encryption with Tempo, refer to Configure TLS communication.

Authentication errors

These errors occur when there are issues with authentication credentials or permissions.

Authentication failed

Error message: 401 Unauthorized or 403 Forbidden

Cause: The authentication credentials are invalid or the user doesn’t have permission to access Tempo. This error appears when you select Save & test or when running a query.

Solution:

Verify that the authentication credentials (username/password or API key) are correct in the data source settings.
Check that the user or service account has the required permissions.
Ensure the authentication method selected in the data source matches what Tempo expects.
For Grafana Cloud, verify that your Cloud Access Policy token has the traces:read scope. Refer to Create a Cloud Access Policy token for setup instructions. To find your stack URL and credentials, refer to Locate your stack URL, user, and password.
For self-managed Tempo, ensure the credentials match those configured in the Tempo server block or reverse proxy.

Save and test returns 404

Error message: Tempo echo endpoint returned status 404 when selecting Save & test

Cause: The Grafana health check reaches the server but the api/echo endpoint returns a 404. The most common cause is incorrect path segments in the URL (such as /api or /tempo on a self-managed instance) that shift the health check request to a nonexistent path. A 404 can also occur when a reverse proxy in front of Tempo rejects requests that include unexpected basic authentication headers.

Solution:

Verify that the URL doesn’t include trailing path segments. For self-managed Tempo, the URL should be http://<HOST>:3200 with no path. Refer to the Connection section for the correct format.
If a reverse proxy sits between Grafana and Tempo, check whether the proxy is returning the 404. Tempo itself ignores unexpected auth headers, but a proxy may reject them.
If Tempo doesn’t require authentication, select No authentication in the data source settings.
Check that the authentication credentials are entered in the correct fields. For Grafana Cloud, the User field is the instance ID (not your Grafana login), and the Password is a Cloud Access Policy token.

Multi-tenant authentication issues

Error message: Unauthorized when using multi-tenant Tempo

Cause: The tenant ID (X-Scope-OrgID header) is missing or incorrect. This header must be set on both the write path (in your Alloy or OpenTelemetry Collector configuration) and the read path (in the Grafana data source settings) to route data to the correct tenant.

Solution:

In the Tempo data source settings, verify that the X-Scope-OrgID header is configured under Authentication > HTTP Headers. Refer to Multi-tenancy for data source configuration steps.
On the write path, verify that the same X-Scope-OrgID header is set in your Alloy or OpenTelemetry Collector configuration so traces are ingested under the correct tenant.
Check that the authenticated user has access to the specified tenant.
For cross-tenant queries, ensure all specified tenants are accessible.

For more information, refer to Enable multi-tenancy and Tenant IDs.

Query errors

These errors occur when there are issues with TraceQL queries or trace lookups.

Trace not found

Error message: Trace not found or trace ID not found

Cause: The specified trace ID doesn’t exist in Tempo or has aged out of the configured retention period. Common reasons include:

Sampling: Head or tail sampling in your instrumentation pipeline (Alloy, OpenTelemetry Collector) may have dropped the trace before it reached Tempo. This is the most common cause of missing traces.
Retention: Traces are only available for the duration set in the Tempo retention configuration. For Grafana Cloud users, default retention periods apply unless you contact Support to change them.
Ingestion failure: The trace may not have been successfully ingested due to rate limits, errors, or misconfigured pipelines.

Solution:

Verify the trace ID is correct and complete.
Check your sampling configuration in Alloy or OpenTelemetry Collector. If head or tail sampling is enabled, the trace may have been intentionally dropped. Refer to Sampling strategies for guidance.
Check that the trace is within the configured retention period for Tempo.
If using time range restrictions, expand the time range in the TraceID query settings.
Enable Use time range in query in the TraceID query settings and adjust the time shift values to search a broader range.
Verify the trace was successfully ingested by Tempo. You can run a broad query to check for recent traces:
traceql
```
{ } | count() > 0
```
If this returns results, traces are being ingested but your specific trace ID may have been dropped by sampling or aged out. For more query examples, refer to the TraceQL cookbook.
For Grafana Cloud users, refer to Troubleshoot TraceQL and search for TraceQL queries that can help investigate missing traces.

TraceQL syntax errors

Error message: parse error or unexpected token

Cause: The TraceQL query contains syntax errors.

Solution:

Verify the query follows TraceQL syntax.
Check that all braces, parentheses, and quotes are balanced.
Ensure attribute names are correctly formatted with the correct scope. Use dot notation for span or resource attributes, for example, span.http.status_code or resource.service.name.
Use the Search query builder to generate valid TraceQL queries if you’re unfamiliar with the syntax.
Verify operators are used correctly. Supported operators include =, !=, =~, !~, >, <, >=, <=.

Query returns no results

Cause: The query is valid but doesn’t match any traces.

Solution:

Expand the time range to search a broader period.
Verify the attribute names and values exist in your traces.
Check that the attribute names match exactly, as they’re case-sensitive.
Use the Search query builder to explore available attributes and values.
Start with a broader query and progressively add filters to narrow results.
Try Grafana Traces Drilldown, a queryless app that lets you explore tracing data using RED metrics without writing TraceQL.
Note that Tempo search is non-deterministic—identical queries can return different results because Tempo scans in parallel and returns the first matching traces. Refer to Understand search behavior for details.
Verify traces are being ingested into Tempo by querying for a known trace ID.

Traces not appearing after successful connection

Cause: Save & test succeeds, but running queries in Explore returns no traces. The data source connection is valid, but no trace data is reaching Tempo, or the queries don’t match any ingested data.

Solution:

Verify that your applications are instrumented and that your collector (Alloy, OpenTelemetry Collector) is forwarding spans to Tempo. Check collector logs for send errors, rejected spans, or drops.
For Grafana Cloud users who send traces with Alloy, refer to Troubleshoot sending traces with Alloy for ingestion-side issues.
Confirm that the query time range covers the period when traces were generated. Trace data only appears for the window in which spans were ingested.
For multi-tenant setups, verify that the X-Scope-OrgID header is the same on both the write path (collector) and the read path (Grafana data source). A mismatch causes traces to be ingested under one tenant and queried under another.
Try a broad query, for example, {}, with a short, recent time range to confirm that any traces exist in Tempo. If this returns results, your original query filters are too narrow.

Query timeout

Error message: context deadline exceeded or query timeout

Cause: The query took too long to execute. Trace searches scan raw trace data across the selected time range, so broad queries over long periods can exceed timeout limits.

Solution:

Reduce the time range to limit the amount of data scanned.
Add more specific filters to narrow the search scope.
Use indexed attributes in your queries when possible for faster lookups.
For self-managed Tempo, increase the query timeout in the Tempo server configuration if appropriate.
If you need aggregate data (rates, counts, durations) rather than individual traces, consider using TraceQL metrics queries (for example, { } | rate()) or pre-computed metrics from the metrics generator, which are optimized for aggregate workloads.

TraceQL metrics query fails

Error message: localblocks processor not found or metrics-generator not configured

Cause: TraceQL metrics queries (such as { } | rate() or { } | count_over_time()) require the local-blocks processor to be active in the Tempo metrics-generator configuration. This processor is separate from the service-graphs and span-metrics processors and must be enabled independently.

Solution:

For self-managed Tempo, activate the local-blocks processor in the overrides configuration. The configuration path depends on your Tempo version and deployment method:
- Standard: overrides.metrics_generator_processors: ["local-blocks"]
- Newer versions and Helm: overrides.defaults.metrics_generator.processors: [local-blocks]
Refer to Configure TraceQL metrics for setup steps and required configuration.
For Grafana Cloud, the local-blocks processor is enabled by default. If you still see this error, contact Grafana Support.
Verify that the metrics generator is running and healthy by checking the Tempo metrics-generator logs for errors.

Streaming issues

Streaming displays TraceQL query results as they become available. Without streaming, you don’t see results until the query completes. Streaming is available for Grafana Cloud users by default. For self-managed Tempo, search streaming requires Tempo v2.2 or later, and metrics streaming requires Tempo v2.7 or later. Both require stream_over_http_enabled: true in the Tempo configuration.

For more information, refer to Streaming.

Streaming not working

Cause: Results don’t appear incrementally and only display after the full query completes.

Solution:

For self-managed Tempo, verify that you’re using the required Tempo version (v2.2 or later for search streaming, v2.7 or later for metrics streaming) and that streaming is enabled in your configuration (stream_over_http_enabled: true). Refer to Tempo gRPC API in the Tempo documentation. Streaming is available for Grafana Cloud users by default.
Verify that Streaming is enabled in your Tempo data source settings. The Search queries and Metrics queries toggles are independent—check that the toggle for the query type you’re using is enabled.
If your Tempo instance is behind a load balancer or proxy that doesn’t support gRPC or HTTP2, streaming may not work. Disable streaming and use standard HTTP queries instead.
If you’re using Private data source connect (PDC) to reach Tempo, verify that the PDC tunnel is running and the Tempo instance is reachable from the PDC agent. PDC connectivity issues can present as streaming timeouts.

gRPC transport security error

Error message: rpc error: code = Unavailable desc = credentials require transport level security

Cause: Grafana tries to use gRPC over an insecure (non-TLS) connection. Streaming uses gRPC, which may require TLS depending on your Tempo configuration.

Solution:

Configure TLS between Grafana and Tempo. Refer to Configure TLS communication.
If TLS isn’t an option, disable streaming in the Tempo data source settings and use standard HTTP queries instead.

Partial results or streaming errors

Error message: stream error or incomplete results during streaming

Cause: The streaming connection was interrupted before all results were delivered.

Solution:

Check network stability between the browser and Grafana.
Verify that proxies or load balancers aren’t terminating long-lived connections.
Disable streaming and use standard queries if issues persist.
Check Tempo server logs for any errors related to streaming.

Service graph issues

The Service Graph visualizes service dependencies and highlights request rate, error rate, and duration (RED metrics) across connections. It requires a linked Prometheus data source with service graph metrics generated by the Tempo metrics generator or Grafana Alloy. For more information, refer to Service Graph and Service Graph view.

Service graph not displaying

Cause: The Service Graph view shows no data or an empty graph.

Solution:

Verify that a Prometheus data source is linked in the Tempo data source Service Graph settings under Additional settings.
For self-managed Tempo, check that the metrics generator is configured to produce service graph metrics. Refer to Service graph metrics in the Tempo documentation.
Ensure the following service graph metrics exist in Prometheus:
- traces_service_graph_request_total
- traces_service_graph_request_failed_total
- traces_service_graph_request_server_seconds_sum
Verify that Grafana Alloy or the Tempo metrics generator is configured to produce these metrics. These are distinct from span metrics (traces_spanmetrics_*), which power the Service Graph view table.
To verify the metrics exist, run the following query in Explore with your Prometheus data source selected:
promql
```
{__name__=~"traces_service_graph_request.*"}
```
Check the Prometheus data source connection is working.
For Grafana Cloud Traces users, refer to Troubleshoot service graph and RED metrics.

Service Graph view table is empty

Cause: The Service Graph node graph renders correctly, but the RED metrics table alongside it shows no data.

Solution:

The Service Graph view table uses a different set of metrics from the node graph. The Grafana frontend queries traces_spanmetrics_calls_total and traces_spanmetrics_latency_bucket. Confirm that these metrics exist in your linked Prometheus data source. Newer Tempo versions may produce traces_spanmetrics_duration_seconds_bucket instead of traces_spanmetrics_latency_bucket. If you only have the newer metric name, the duration column shows no data until the Grafana frontend supports the renamed metric.
Verify that span metrics generation is enabled in your Tempo or Alloy configuration. Refer to Enable service graphs in the Tempo documentation.

High cardinality warnings

Cause: Service graph or span metrics queries produce high-cardinality warnings or slow queries.

Solution:

Review the label cardinality of your service graph and span metrics. High-cardinality labels (such as unique Pod names or request IDs) create many time series and degrade query performance.
Refer to Cardinality in the Tempo documentation for guidance on managing label cardinality.

Service graph shows incomplete data

Cause: Some services or connections are missing from the graph.

Solution:

Verify that all services are instrumented and sending spans.
Check that span names and service names are consistent.
Ensure the time range includes data from all expected services.
Verify the Prometheus data source has metrics for all services.

Trace to logs/metrics/profiles issues

These issues relate to the correlation features that link traces to other telemetry signals. For configuration details, refer to Trace to logs, Trace to metrics, and Trace to profiles.

Trace to logs link not appearing

Cause: Links to logs don’t appear when viewing a trace.

Solution:

Verify that a Loki or other log data source is configured in the Trace to logs settings.
Check that the configured tags exist in your spans.
Ensure at least one mapped tag has a non-empty value in the span.
If using a custom query, verify the query syntax is valid for the target data source.
Check that Filter by trace ID or Filter by span ID aren’t enabled when using a custom query.
If using Filter by trace ID with Loki derived fields, verify that the derived field regular expression matches the trace ID field name used in your logs. Different instrumentation libraries use different field names (traceID, trace_id, or traceId). Refer to Trace to logs for common regular expression patterns by instrumentation format.

Loki log lines don’t show a trace link

Cause: The reverse direction—navigating from a Loki log line to a trace—isn’t working. No trace link appears on log lines.

Solution:

Verify that a derived field is configured in the Loki data source under Additional settings > Derived fields. The derived field extracts the trace ID from log lines and creates a link to Tempo.
Check that the derived field regular expression matches the trace ID format in your logs. Use Show example log message in the derived field settings to test the regular expression against a sample log line.
Confirm the Internal link toggle is enabled and points to the correct Tempo data source.
If different services use different trace ID field names, you need a separate derived field entry for each format. Refer to Trace to logs for regular expression patterns by format.

Trace to metrics link not appearing

Cause: Links to metrics don’t appear when viewing a trace.

Solution:

Verify that a Prometheus or other metrics data source is configured in the Trace to metrics settings.
Check that the configured tags exist in your spans.
If using custom queries, verify they’re valid PromQL.
Ensure the $__tags variable is correctly placed in custom queries.

Trace to profiles link not appearing

Cause: Links to profiling data don’t appear when viewing a trace.

Solution:

Verify that a Grafana Pyroscope data source is configured in the Trace to profiles settings.
Check that the configured tags exist in your spans.
Verify that the configured profile type matches what Pyroscope has data for (for example, process_cpu:cpu:nanoseconds:cpu:nanoseconds). A mismatch between the configured profile type and the available data causes the link to not appear.
Tighten the Span start time shift and Span end time shift values. Profiles are sampled at intervals, so a narrow time window around the span is more likely to match profile data than a wide one.
Verify that the profiling data source is connected and returning data.

Links appear but return no data

Cause: The generated query doesn’t match any data in the target data source.

Solution:

Adjust the Span start time shift and Span end time shift to widen the time range.
Verify that log or metric labels match the span attributes.
Check that the tag mappings correctly translate attribute names between data sources. Span attributes use dots (for example, service.name) but Loki labels use underscores (for example, service_name). Ensure tag mappings account for this difference. Refer to Trace to logs for tag mapping configuration.
If a tag like pod is stored as Loki structured metadata rather than an indexed label, the auto-generated stream selector {pod="..."} returns no results. Enable Use custom query and move the tag to a pipeline filter. Refer to Trace to logs for a custom query example.
Run the generated query directly in the target data source’s Explore view to confirm it returns data outside of the trace context.
Use the Query Inspector to view the generated query and verify it’s correct.

Performance issues

These issues relate to slow queries or high resource usage.

Slow trace queries

Cause: Queries take a long time to return results.

Solution:

Reduce the dashboard or Explore time range.
Add more specific filters to narrow the search scope.
Use indexed attributes in TraceQL queries when possible.
Enable streaming to see results as they become available.
Consider using the Search query builder which can help optimize queries.

High memory usage during queries

Cause: Large traces or result sets consume excessive memory.

Solution:

Reduce the Limit setting in query options to return fewer traces.
Reduce the Span Limit setting to return fewer spans per trace.
Add filters to reduce the result set size.
Avoid querying very large time ranges.

Other common issues

The following issues don’t produce specific error messages but are commonly encountered.

Node graph not displaying

Cause: The node graph visualization doesn’t appear above the trace view.

Solution:

Verify that Node graph is enabled in the Tempo data source settings under Additional settings.
Check that the trace contains multiple spans with parent-child relationships. Spans must include traceID and spanID, and non-root spans must include parentSpanID to establish hierarchy. If these identifiers are missing, the trace data is incomplete at the instrumentation level and the graph can’t be rendered.

Autocomplete not working

Cause: The TraceQL editor doesn’t show suggestions for attributes or values. Autocomplete requires Tempo to enumerate tags, which requires a supported storage backend (vParquet or later).

Solution:

For self-managed Tempo, verify that the storage backend supports tag search. Tempo requires vParquet format or later for tag-based search and autocomplete. Refer to Parquet configuration for details. Grafana Cloud Traces supports autocomplete by default.
Check the Tags time range setting in the data source settings to ensure it covers a period with recent trace data.
Increase the Tag limit setting if you have many unique attributes.
Verify the Tempo data source connection is working by selecting Save & test in the data source settings.

Data source settings disappear after restart

Cause: You configured settings in the Grafana UI (such as trace-to-logs links, service graph, or span time shifts), but they disappear after Grafana restarts. This happens when the data source is provisioned via a YAML configuration file, Helm chart, or Terraform. Provisioned data sources are read-only in the UI—any changes you make through the UI aren’t persisted and silently revert on the next restart.

You can tell a data source is provisioned if the settings form is read-only and the button reads Test instead of Save & test.

Solution:

To change settings permanently, edit the provisioning YAML file and restart Grafana (or wait for the provisioning system to reload). Any setting available in the UI can be set in the YAML file, including trace-to-logs, trace-to-metrics, and service graph configuration. Refer to Provision the Tempo data source for the full YAML reference.
For Grafana Cloud, you can clone the provisioned data source to create an editable copy that persists UI changes. Refer to Clone a provisioned data source for Grafana Cloud for the steps.

TraceQL alerting not available

Cause: You can select the Tempo data source in the Grafana Alerting rule builder, but standard TraceQL search queries return trace data (spans and trace IDs), not numeric time series. Alert rules require numeric data to evaluate thresholds, so trace search queries don’t produce usable alert conditions.

Solution:

TraceQL metrics queries (such as { } | rate() or { status = error } | count_over_time()) return numeric time series and can drive alert thresholds directly. However, TraceQL alerting requires the experimental tempoAlerting feature flag, which is disabled by default. To enable it, set the tempoAlerting feature flag in your Grafana configuration. For Grafana Cloud, contact Support to enable it. For step-by-step setup, refer to Trace-based alerts.

Caution
The tempoAlerting feature flag is experimental. For production alerting, consider the Prometheus-based approach described below.

Alternatively, use the Tempo metrics generator to produce Prometheus metrics from trace data, then create alert rules against the Prometheus data source that stores those metrics.

Configure the Tempo metrics generator (using the service-graphs, span-metrics, or local-blocks processor) to produce metrics from traces and remote-write them to a Prometheus-compatible backend.
Create alert rules against the Prometheus data source that receives those metrics. For example, alert on high error rates using traces_spanmetrics_calls_total with a status_code="STATUS_CODE_ERROR" filter.
Refer to the feature support table in the Tempo data source overview for details on alerting support.

Switching between multiple Tempo instances

Cause: You have multiple Tempo data sources configured (for example, one per environment or per tenant), but there’s no built-in UI dropdown in Explore or dashboards to switch between them dynamically.

Workaround:

In dashboards, create a data source template variable filtered to tempo type. This adds a dropdown that lets viewers switch between Tempo instances without editing the dashboard.
In Explore, use the data source picker in the top bar to switch between configured Tempo data sources.
For multi-tenant setups on a single Tempo instance, consider using the X-Scope-OrgID header instead of separate data sources. Refer to Multi-tenancy for configuration details.

Private data source connect issues (Grafana Cloud only)

These issues relate to connecting to a private Tempo instance using Private data source connect (PDC). PDC is only available in Grafana Cloud.

Note
Private data source connect allows Grafana Cloud to securely connect to data sources in your private network without exposing them to the public internet. For setup instructions, refer to Private data source connect.

PDC agent not connected

Error message: No healthy PDC agents available or Private data source connect agent not connected

Cause: The PDC agent installed in your private network isn’t running or can’t connect to Grafana Cloud.

Solution:

Verify the PDC agent is running in your private network.
Check the PDC agent logs for connection errors.
Ensure the PDC agent has outbound network access to Grafana Cloud endpoints.
Verify the PDC agent is using the correct token for your Grafana Cloud instance.
Check that firewall rules allow outbound HTTPS (port 443) connections from the PDC agent.
Restart the PDC agent if it appears stuck or unresponsive.

For more information, refer to Private data source connect.

Connection through PDC failing

Error message: Failed to connect through private data source or Connection refused via PDC

Cause: The PDC agent is connected to Grafana Cloud but cannot reach the Tempo instance in your private network.

Solution:

Verify the Tempo URL is correct and accessible from the machine running the PDC agent.
Check that the PDC agent can resolve the Tempo hostname (DNS resolution).
Ensure there are no firewall rules blocking connections from the PDC agent to Tempo.
Verify the Tempo port (default 3200) is open and accessible.
Test connectivity from the PDC agent host to Tempo using curl or telnet.

PDC timeout errors

Error message: Request timed out via private data source connect

Cause: Queries through the PDC tunnel are timing out.

Solution:

Check network latency between the PDC agent and Tempo.
Verify the PDC agent host has sufficient resources (CPU, memory, network bandwidth).
Ensure no network devices between the PDC agent and Tempo are dropping long-running connections.
For large queries, consider reducing the time range or adding more specific filters.
Check if multiple data sources are sharing the same PDC agent, which may cause resource contention.

PDC authentication mismatch

Error message: Unauthorized or 403 Forbidden when using PDC

Cause: The authentication configured in the data source doesn’t work with the PDC connection.

Solution:

Verify that authentication credentials are configured correctly in the Grafana Cloud data source settings.
Check that the Tempo instance accepts the authentication method configured.
Ensure any required headers (such as X-Scope-OrgID for multi-tenant Tempo) are properly forwarded through PDC.
Test the connection directly from the PDC agent host to verify credentials work outside of PDC.

Get additional help

If you continue to experience issues after following this troubleshooting guide:

For Grafana Cloud users, try Grafana Assistant to investigate trace issues interactively (if enabled on your account).
For Grafana Cloud users who send traces with Alloy, refer to the Alloy troubleshooting steps for ingestion-side issues.
Ask questions in the Grafana community forums or the Grafana Community Slack.
Review the Grafana GitHub issues for known bugs.
Review the Tempo GitHub issues for Tempo-specific bugs.
Enable debug logging in Grafana to capture detailed error information.
Check Tempo server logs for additional error details.
Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Contracted user.

When reporting issues, include:

Grafana version
Tempo version
Error messages (redact sensitive information)
Steps to reproduce
TraceQL query examples (redact sensitive data)
Relevant configuration settings