Version 2.1 release notes

The Tempo team is pleased to announce the release of Tempo 2.1.

With this release, Tempo enhances TraceQL so you can select traces using features such as span kind, aribtrary arithmetic, and new aggregates min/max/sum. In addition, Tempo 2.1 introduces vParquet2, a new experimental Parquet version designed to be more compatible with other Parquet implementations.

Note: For a complete list of changes and enhancements, refer to the Tempo 2.1 changelog.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.1 are highlighted below.

Enhancements to TraceQL

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. TraceQL makes it possible to find traces that otherwise would have been difficult or even impossible to identify with Tempo’s 1.x search.

With this release, we’ve expanded the TraceQL language. In Tempo 2.0, TraceQL allowed users to select traces based upon resource attributes, time, duration, and basic aggregates. In Tempo 2.1, users can also select traces based on the following:

Enable querying by kind, an intrinsic attribute (PR 2217)
- { kind = server }, { kind = client }, { kind = consumer }
- { kind = server && duration > 1s }
Support arithmetic expressions (PR 2146)
- { span.bytesProcessed > 10 * 1024 * 1024 }
- { span.bytesProcessed / 1024 / 1024 > 10 }
- { span.jobs / span.bytes > 3 }
Enable min, max, and sum aggregate filters
- {} | min(duration) > 1s
- {} | max(span.bytesProcessed) < 1024 * 1024
- {} | sum(duration) > 1h
Support for more numeric types: Compare floating point numbers and integers, allow floating point numbers for durations, and support negative numbers when doing aggregate filters. (PR 2139, 2304, 2289)
- { 1 = 1.0 }
- { duration > 1.5s }, {duration > 1s500ms }
- {} | count() > -1

This update also includes major performance improvements to TraceQL query performance. [PR 2119]

To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, see future work.

TraceQL query editor improvements (experimental)

Starting in Grafana 9.5, users have access to a Tempo Search tab powered by TraceQL. To enable this feature, use Grafana’s traceqlSearch feature flag. The editor can autocomplete queries and has a streamlined interface.

Grafana will soon include a new Search screen that can build a TraceQL query based upon options selected in drop-downs. Contact Support to enable this feature in Grafana Cloud.

For more information, refer to the TraceQL query editor documentation.

vParquet2 (experimental)

Tempo 2.1 offers an experimental update to the Parquet backend: vParquet2. Due to the dedicated span duration column, there is a performance gain for TraceQL queries containing duration. vParquet2 will also be a prerequisite for TraceQL’s structural operators, which will be introduced in a future release. [PR 2244]

You can select the block format Tempo uses by changing the block format version option. To use the new format, set the block format to vParquet2. For more information, read the Parquet configuration documentation.

The new encoding implements Parquet schema changes that are not compatible with vParquet:

Renamed columns to make the schema more consistent and closer to names and terminologies used by OTEL
Add a dedicated span column for duration
Additional columns that will enable new TraceQL features, including hierarchical queries

Other enhancements and features

For a complete list, refer to the Tempo 2.1 changelog.

PR 2105 Log when a trace is too large to compact. Traces are dropped when they exceed a certain limit set in the ingesters. Any dropped traces are now logged with the number of dropped spans and the trace ID.
PR 2195 Add support for Azure Workload Identity authentication
PR 2130 Add command to tempo-cli to migrate a tenant
PR 2093 Add support for s3 session token in static configuration

Upgrade considerations

When upgrading to Tempo 2.1, be aware of these breaking changes.

Breaking changes to metric names exposed by Tempo

All Prometheus metrics exposed by Tempo on its /metrics endpoint that were previously prefixed with cortex_ have now been renamed to be prefixed with tempo_ instead. (PR 2204)

Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR 2008)

The query_frontend_result_metrics_inspected_bytes metric was removed in favor of query_frontend_bytes_processed_per_second.

Remove support for Search on v2 blocks

Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section:

overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:

The following metrics configuration was also removed:

tempo_ingester_trace_search_bytes_discarded_total

This change means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. One way to avoid this is to upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces. Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable.

Finally, parquet files are no longer cached when carrying out searches (PR 2164).

Bug fixes

For a complete list, refer to the Tempo 2.1 changelog.

2.1.1

PR 2400 Fixed issue where Tempo sometimes flips booleans from false->true at storage time.

2.1

PR 2220 Correctly connect context during compaction
PR 2081 Apply rate() to the bytes/s panel in the tenant’s dashboard.
PR 2111 Retry copy operations during compaction in GCS backend.
PR 2164 Improve locking and search head block in SearchTagValuesV2
PR 2152 Fix not closing WAL block file before attempting to delete the folder.
PR 2219 Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like status.
PR 2254 Correctly return unique spans when &&ing and ||ing spansets.