Query tracing data on Grafana Labs

Search traces using the query builder

Tue, 14 Apr 2026 14:59:56 +0000

Search traces using the query builder

The Search query builder lets you create TraceQL queries using drop-down lists and text fields instead of writing syntax directly. Each selection you make generates a TraceQL query behind the scenes. You can view the generated query and copy it to the TraceQL editor at any time.

Use Search when you’re exploring data or learning TraceQL patterns. For complex queries involving structural operators, aggregations, or features that Search doesn’t support, use the TraceQL editor. To learn the full query syntax, refer to Construct a TraceQL query.

If queries return no results, check that your Tempo data source is configured and connected.

Enable Search with the query builder

This feature is automatically available in Grafana 10 (and newer) and Grafana Cloud.

To enable the TraceQL query builder in self-managed Grafana through version 10.1, enable the traceqlSearch feature toggle.

Write TraceQL queries using Search

The Search query builder, located on the Explore > Query type > Search in Grafana, provides drop-down lists and text fields to help you write a query. The selections you make automatically generate a TraceQL query.

The builder lets you run the most common queries in as few clicks as possible. You don’t need to know the underlying query language or database architecture to use it.

The builder supports a subset of TraceQL capabilities, including some structural operators (>>, >, =~, !=).

To access Search, select your Tempo data source, and then choose Explore and select Query type > Search. You can use the query builder to search trace data by resource service name, span name, duration, one or more tags. The examples on this page use the default filters.

In addition, you can add query builder blocks, view the query history, and use the Inspector to see details.

Perform a search

To perform a search, you need to select filters and then run the query. The results appear underneath the query builder. The screenshot identifies the areas used to perform a search.

Number	Name	Action	Comment
1	Data source	Use the data source drop-down list to select a Tempo data source.	Each data source has its own version of search. This Search is specific to the Tempo data source.
2	Query type	Select Search.
3	Choose filter	Choose one or more of the filters.	Optional. You can execute an empty query in the Search tab. In TraceQL, `{}` is a valid query and is the default query to provide a list of all traces or spans.
4	Filters conditions	Select options for one or more filters. For example, you can define a filter where Service Name (`resource.service.name`) equals (`=`) `user`.	Optional. At least one tag or filter must be defined.
5	Tags and Aggregate by	Add tags for span, resource, or unscoped and define their conditions. Use Aggregate by to group results.	Optional.
6	TraceQL query	Displays the TraceQL query constructed by your selections.	This TraceQL query is executed when you select Run query.

Every query searches the data for the selected time frame. By default, queries run against data from the last hour. Select Time range to the left of Run query to choose the time range for the data your query runs against. Read the dashboard time range documentation to learn more.

To access the Search query builder, use the following steps:

Sign into Grafana.
Select your Tempo data source.
From the menu, choose Explore and select Query type > Search.

Define filters

Using filters, you refine the data returned from the query by selecting Service Name, Span Name, Status, or Duration. Duration represents span time, calculated by subtracting the end time from the start time of the span.

Grafana administrators can change the default filters using the Tempo data source configuration. Filters can be limited by the operators. The field type determines the available operators. For example, Span Name and Service Name are string fields so the comparison operators are equals (=), not equal (!=), matches regular expressions (=~), or doesn’t match regular expression (!~). Duration is a duration field type and uses range selections (>, >=, <, <=).

When you select multiple values for the same filter, Grafana automatically changes the operator to the regular expression (regex) operator =~ and concatenates the values with a |. This capability only applies to fields with drop-down value selection.

For example, if you choose Span Name = get and then Span Name = log_results_cache, operator drop-down list changes from = to =~ and both get and log_results_cache are listed in the Span Name field. The resulting query is updated with this:

{duration>5ms && duration<10ms && name=~"get|log_results_cache"}

To define filters, follow these steps:

Choose one of the filters.
Select a comparison operator from the drop-down.
Service Name, Span Name, and Status only: Select one or more values from the drop-down.
Duration only: Enter values and units for the range and choose comparison operators for the drop-downs. Units can be nanoseconds (ns), milliseconds (ms), seconds (s), minutes (m), and hours (h).

You can either select Run query to execute the query or define tags and then run the query.

Define tags

You can add any tags to your query to further filter the results. Tags can be selected by scoped (span or resource) or unscoped. If you select unscoped, then all tags are searched for matches.

To add a tag, follow these steps:

Select span, resource, or unscoped.
Select a tag from the Select tag drop-down.
Select a comparison operator.
Select a value from the Select value drop-down. This field is populated based upon the tag.
Optional: Select + to add another tag.

Optional: Use Aggregate by

Warning
Metrics summary API and the Aggregate by feature are deprecated in Grafana Cloud and Grafana 11.3 and later. It will be removed in a future release.

Using Aggregate by, you can calculate RED metrics (total span count, percent erroring spans, and latency information) for spans of kind=server that match your filter criteria, grouped by one or more attributes. This capability is based on the metrics summary API. Metrics summary only calculates summaries based on spans received within the last hour. For additional information, refer to Traces to metrics: Ad-hoc RED metrics in Grafana Tempo with Aggregate by.

Aggregate by is disabled by default. Enable the metricsSummary feature toggle in Grafana to use this feature.

Your Grafana Tempo data source must also point to a Tempo database with the Metrics Summary API enabled. Contact Grafana Support to enable this feature in Grafana Cloud.

Use Aggregate by

When you use Aggregate by, the selections you make determine how the information is reported in the Table. Every combination that matches selections in your data is listed in the table. Each aggregate value, for example intrinsic:name, has a corresponding column in the results table.

For example, names matching GET /:endpoint with a span.http.user_agent of k6/0.46 appeared in 31,466 spans. Instead of being listed by traces and associated spans, the query results are grouped by the selections in Aggregate by.

The RED metrics are calculated for every name and user agent combination found in your data.

The screenshot shows all of the successful HTTP status_code API calls against the mystical-server service. The results are shown in the same order used in Aggregate by. For example, Aggregate by lists intrinsic.name followed by span.http.user_agent. The first column in the results Table shows name and then span.http.user_agent.

To use this capability:

In the Aggregate by row, select a scope from the first drop-down box. For example, span.
Select an attribute from the second drop-down.
Optional: Select + to add an Aggregate by row.
Optional: Select a Time range to expand or narrow the data set for an hour’s range.
Select Run query.

Optional: Add query and service graph blocks

Using Add query, you can have successive query or service node blocks that run in sequential order. For example, query A runs and then query B. You can reorder the queries by dragging and dropping them above or below other queries. Select + Add query to add another query block.

For more information, refer to Use query types together.

Run queries and view results

Select Run query to run the TraceQL query (1 in the screenshot).

Queries can take a little while to return results. The results appear in a table underneath the query builder. Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot). Span Filters (4 in the screenshot) provide an additional to refine the query results.

Stream results

The Tempo data source supports streaming responses to TraceQL queries so you can see partial query results as they come in without waiting for the whole query to finish.

When active, all configured Tempo data sources attempt to use streaming. You can activate and control which Tempo data sources do and don’t attempt to stream results at the per-data source level using the Streaming section of the Tempo data source configuration. For more information, refer to the Tempo data source documentation.

Streaming is available for both the Search and TraceQL query types. You’ll get immediate visibility of incoming traces on the results table.

Use filters and tags on spans

Using Span Filters, you can use filters to refine results when viewing span details. These filters are available when viewing details for a trace.

You can continue to apply filters until you have narrowed down your resulting spans to the select few you are most interested in.

Service Name, Span Name, Duration, and Tags have the same function and operation as the filters of the same name in the Search query builder.

In addition, you can search for a keyword, opt to Show matches only, opt to Show critical path only, and browse matches using Prev and Next.

Use Clear to reset the filters.

Try these common searches

The following examples show what to select in the Search builder and the TraceQL query each selection generates. Adjust service names and values to match your environment.

Find error spans for a service:

Set Service Name to your service (for example, frontend).
Set Status to error.

Generated query:

{ resource.service.name = "frontend" && status = error }

Find slow API calls:

Set Span Name to the operation (for example, POST /api/orders).
Set Duration to > 500ms.

Generated query:

{ name = "POST /api/orders" && duration > 500ms }

Filter by HTTP status code:

Select Add tag, choose span, and select http.response.status_code.
Set the operator to >= and the value to 500.

Generated query:

{ span.http.response.status_code >= 500 }

For more query examples, refer to TraceQL query examples.

Next steps

TraceQL query examples: Copy-paste query examples for common use cases
Write TraceQL queries with the editor: For complex queries the Search builder doesn’t support
Construct a TraceQL query: Full TraceQL syntax reference
Service Graph and Service Graph view: Visualize service dependencies
Span filters: Refine results in the trace detail view

Write TraceQL queries with the editor

Tue, 14 Apr 2026 14:59:56 +0000

Write TraceQL queries with the editor

Use the TraceQL editor when you need structural queries across parent and child spans, aggregations, or other features that the Search query builder doesn’t support. TraceQL queries follow the pattern { conditions } | pipeline. Refer to Construct a TraceQL query for the full syntax.

To get started, paste this query into the editor and select Run query:

{ resource.service.name = "frontend" && span:status = error }

This returns all error spans from the frontend service. Replace frontend with your service name. For more examples, refer to TraceQL query examples.

If queries return no results, check that your Tempo data source is configured and connected.

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what is searched.

To learn more about how to query by TraceQL, refer to the TraceQL documentation.

The TraceQL query editor in Grafana Explore lets you search by trace ID and write TraceQL queries using autocomplete.

Before you begin

This feature is automatically available in Grafana 10 (and newer) and Grafana Cloud.

To use the TraceQL query editor in self-hosted Grafana 9.3.2 and older, you need to enable the traceqlEditor feature toggle.

Streaming and gRPC

If you’re trying to query a self-managed Grafana Tempo or Grafana Enterprise Traces database with a gateway, such as nginx, in front of it from your hosted Grafana, that gateway (for example, nginx) must allow gRPC connections. If it doesn’t, streaming won’t work and queries will fail to return results.

If you can’t configure your gateway to allow gRPC, deactivate streaming in your hosted Grafana. In Grafana 11.2 and newer, you can deactivate the Streaming option in your Tempo data source settings from Connections > Data sources in the Grafana main menu. You can also open a support escalation to request streaming query results be disabled in your hosted Grafana.

Write TraceQL queries using the query editor

The Tempo data source’s TraceQL query editor helps you query and display traces from Tempo in Explore.

To access the query editor, follow these steps:

Sign into Grafana or Grafana Cloud.
Select Explore from the main menu.
Select a Tempo data source.
Select the TraceQL tab.
Start your query on the text line by entering {. For help with TraceQL syntax, refer to the Construct a TraceQL query documentation.

Optional: Select Copy query from Search to transfer a builder query to the editor.
Optional: Use the Time picker drop-down list to change the time and range for the query (refer to the documentation for instructions).
Once you’ve finished your query, select Run query.

This video provides an example of creating a TraceQL query using the custom tag grouping.

Query by TraceID

To query a particular trace by its trace ID:

From the menu, choose Explore, select the desired Tempo data source, and navigate to the TraceQL tab.
Enter the trace ID into the query field. For example: 41928b92edf1cdbe0ba6594baee5ae9
Click Run query or use the keyboard shortcut Shift + Enter.

Use autocomplete to write queries

You can use the query editor’s autocomplete suggestions to write queries. The editor detects spansets to provide relevant autocomplete options. It uses regular expressions (regex) to detect where it’s inside a spanset and provide attribute names, scopes, intrinsic names, logic operators, or attribute values from the Tempo API, depending on what’s expected for the current situation.

Anchored regular expressions

Regular expressions are anchored at both ends. This anchoring makes the queries faster and matches the behavior of PromQL, where regular expressions are also fully anchored.

An unanchored query, such as: { span.foo =~ "bar" } is now treated as: { span.foo =~ "^bar$" }.

If you use TraceQL with regular expressions in your Grafana dashboards and you want the unanchored behavior, update the queries to use the unanchored version, such as { span.foo =~ ".*bar.*"}.

Create a query with autocomplete

To create a query using autocomplete, follow these steps:

From the menu, choose Explore, select the desired Tempo data source, and navigate to the TraceQL tab.
Enter your query. As you type your query, autocomplete suggestions appear as a drop-down. Each letter you enter refines the autocomplete options to match.
Use your mouse or arrow keys to select any option you wish. When the desired option is highlighted, press Tab on your keyboard to add the selection to your query.
Once your query is complete, select Run query.

View query results

Query results appear in a table, such as Table - Traces, under the query editor. Each span (and the trace it belongs to) matching the query conditions is returned by the query. If there are no filter conditions, all spans are matching and thus returned with their associated traces.

A query is performed against a defined time interval, relative (for example, the last 3 hours) or absolute (for example, from X date-time to Y date-time). The query response is also limited by the number of traces (Limit) and spans per spanset (Span Limit).

TraceQL query editor
Query options: Limit, Span Limit and Table Format (Traces or Spans).
Trace (by Trace ID). The Name and Service columns are displaying the trace root span name and associated service.
Spans associated with the Trace.

Selecting the trace ID from the returned results opens a trace diagram. Selecting a span from the returned results opens a trace diagram and reveals the relevant span in the trace diagram.

For more information on span details, refer to Traces in Explore.

Querying spansets with a large number of spans can negatively impact performance. You can use the Span Limit field in Options section of the TraceQL query editor. This field sets the maximum number of spans to return for each span set. By default, the maximum value that you can set for the Span Limit value (or the spss query) is 100. In Tempo configuration, this value is controlled by the max_spans_per_span_set parameter and can be modified by your Tempo administrator. Entering a value higher than the default results in an error.

Note
Changing the value of max_spans_per_span_set isn’t supported in Grafana Cloud.

Focus on traces or spans

Under Options, you can choose to display the table as Traces or Spans focused.

When the Table Type option is set to Spans, the traces and spansets are flattened into a list of spans. The trace service and trace name are added to the row of each span to add context.

Using the Spans option makes it easier access the spans to apply transformations and plot them in dashboards.

Stream results

The Tempo data source supports streaming responses to TraceQL queries so you can see partial query results as they come in without waiting for the whole query to finish.

Streaming is available for both the Search and TraceQL query types, and you’ll get immediate visibility of incoming traces on the results table.

To learn how to activate streaming, refer to Streaming in the Tempo data source documentation.

Next steps

TraceQL query examples: Copy-paste query examples for common use cases
Search traces using the query builder: Build queries visually
Construct a TraceQL query: Full TraceQL syntax reference
Service Graph and Service Graph view: Visualize service dependencies
Span filters: Refine results in the trace detail view

TraceQL query examples

Tue, 14 Apr 2026 14:59:56 +0000

TraceQL query examples

Use these TraceQL queries in Grafana Explore with a Tempo data source. Adjust service names, routes, and attribute values to match your environment.

These examples use OpenTelemetry semantic conventions. If your instrumentation uses different attribute names, substitute them in the queries.

For the full TraceQL syntax, refer to Construct a TraceQL query. For an extended recipe collection, refer to the TraceQL cookbook for Grafana Cloud Traces.

Tip
If a query returns no results, widen the time range and confirm the correct Tempo data source is selected. Prefer trace-level intrinsic fields like trace:duration and trace:rootService for faster queries.

Find error spans

Filter by error status or HTTP error codes:

{ status = error }

{ span.http.status_code >= 500 }

{ span.http.status_code > 399 }

Find slow requests

Filter by span duration. Common thresholds are 1 second and 5 seconds:

{ duration > 1s }

{ duration > 5s }

Filter a specific endpoint by duration:

{ span.http.url = "/api/checkout" && duration > 2s }

Find exceptions by type

Query exception events recorded on spans. Exception data uses the event scope because OpenTelemetry records exceptions as span events:

{ event.exception.message =~ "context cancelled" }

{ event.exception.type = "NotFoundException" }

Find error spans that have an exception message:

{ status = error && event.exception.message != "" }

Filter by service

Select spans from a specific service or match services by pattern. You can use the Service Graph view to identify which services have high error rates or latency, then query those services with TraceQL.

{ resource.service.name = "checkout" }

{ resource.service.name =~ "payment.*" && status = error }

Find errors and slow spans together

Combine error status and duration filters to identify likely root causes:

{ resource.service.name = "api" && status = error && duration > 1s }

Analyze service dependencies

Find errors on a specific API path:

{ span.http.url =~ ".*/api/.*" && span.http.status_code >= 500 }

Use the descendant operator (>>) to find traces where a frontend service calls a downstream service that errors:

{ resource.service.name = "frontend" } >> { resource.service.name = "database" && status = error }

Query rate and count metrics

Run these in Metrics mode in Explore. TraceQL metrics queries have a default 24-hour time-range limit:

{ } | rate()

{ status = error } | rate()

{ resource.service.name = "api" } | count_over_time()

{ duration > 1s } | rate()

Filter by region or environment

Narrow results to a specific cloud region or deployment environment:

{ resource.cloud.region = "us-east-1" && status = error }

{ resource.deployment.environment = "production" && duration > 2s }

Use dashboard variables

Replace hard-coded values with dashboard variables when using these queries in panels. For example, using a $service variable:

{ resource.service.name = "$service" && status = error }

More resources

TraceQL cookbook for Grafana Cloud Traces - Extended recipe collection with additional examples
Construct a TraceQL query - Full TraceQL syntax, scopes, and operators
Query tracing data - Query editor modes and options
Service Graph and Service Graph view - Visualize service dependencies