Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

DocumentationGrafana documentationData sourcesTempoQuery editor

Tempo query editor

The Tempo data source’s query editor helps you query and display traces from Tempo in Explore.

This topic explains configuration and queries specific to the Tempo data source. For general documentation on querying data sources in Grafana, see Query and transform data.

Query by search

Use this to search for traces by service name, span name, duration range, or process-level attributes that are included in your application’s instrumentation, such as HTTP status code and customer ID.

To configure Tempo and the Tempo data source for search, refer to Configure the data source.

To search for traces:

1. Select Search from the Query type selector.
2. Fill out the search form:

Search recent traces

You can search recent traces held in Tempo’s ingesters. By default, ingesters store the last 15 minutes of tracing data.

To configure your Tempo data source to use this feature, refer to the Tempo documentation.

Search the backend datastore

Tempo includes the ability to search the entire backend datastore.

To configure your Tempo data source to use this feature, refer to the Tempo documentation.

Query by TraceID

To query a particular trace:

1. Select the TraceQL query type.
2. Enter the trace’s ID into the query field.

Query by TraceQL

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces. The default traces search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. Query results are returned faster because the queries limit what is searched

To learn more about how to query by TraceQL, refer to the TraceQL documentation.

You can create TraceQL queries using the Query editor or using Search query tab (preview feature).

Create TraceQL queries using Search

Using the Search tab in Explore, you can use the query builder’s drop-downs to compose TraceQL queries. The selections you make automatically generate a TraceQL query.

To access Search, select your Tempo data source, and then choose Explore and select Query type > Search. You can use the query builder to search trace data by resource service name, span name, duration, and one or more tags. The examples on this page use the default filters.

In addition, you can add query builder blocks, view the query history, and use the Inspector to see details.

Perform a search

To perform a search, you need to select filters and/or tags and then run the query. The results appear underneath the query builder. The screenshot below identifies the areas used to perform a search.

Every query searches the data for the selected time frame. By default, queries run against data from the last hour. Select Time range to the left of Run query to choose the time range for the data your query runs against. Read the dashboard time range documentation to learn more.

To access Search, use the following steps:

1. Sign into Grafana.
2. Select your Tempo data source.
3. From the menu, choose Explore and select Query type > Search.

Define filters

Using filters, you refine the data returned from the query by selecting Resource Service Name, Span Name, or Duration. The Duration represents span time, calculated by subtracting the end time from the start time of the span.

Grafana administrators can change the default filters using the Tempo data source configuration. Filters can be limited by the operators. The available operators are determined by the field type. For example, Span Name and Resource Service Name are string fields so the comparison operators are equals (=), not equal (!=), or regular expressions (=~). Duration is a duration field type and uses range selections (>, >=, <, <=).

When you select multiple values for the same filter, Grafana automatically changes the operator to the regex operator =~ and concatenates the values with a |. This capability only applies to fields with drop-down value selection.

For example, if you choose Span Name = get and then Span Name = log_results_cache, operator drop-down changes from = to =~ and both get and log_results_cache are listed in the Span Name field. The resulting query is updated with this:

{duration>5ms && duration<10ms && name=~"get|log_results_cache"}

To define filters, follow these steps:

1. Choose one of the filters.
2. Select a comparison operator from the drop-down.
3. Resource Service Name and Span Name only: Select one or more values from the drop-down.
4. Duration only: Enter values and units for the range and choose comparison operators for the drop-downs. Units can be nanoseconds (ns), milliseconds (ms), seconds (s), minutes (m), and hours (h).

You can either select Run query to execute the query or define tags and then run the query.

Define tags

You can add any tags to your query. Tags can be selected by scoped (span or resource) or unscoped. If you select unscoped, then all tags are searched for matches.

To add a tag, follow these steps:

1. Select span, resource, or unscoped.
2. Select a tag from the Select tag drop-down.
3. Select a comparison operator.
4. Select a value from the Select value drop-down. This field is populated based upon the tag.
5. Optional: Select + to add an additional tag.

Optional: Add queries

Using Add query, you can have successive queries that run in sequential order. For example, query A runs and then query B. You can reorder the queries by dragging and dropping them above or below other queries. Select + Add query to add another query block.

Run queries and view results

Select Run query to run the TraceQL query (1 in the screenshot).

Queries can take a little while to return results. The results appear in a table underneath the query builder. Selecting a Trace ID (2 in the screenshot) displays more detailed information (3 in the screenshot).

Query Loki for traces

To find traces to visualize, you can use the Loki query editor. For results, you must configure derived fields in the Loki data source that point to this data source.