Grafana Tempo 2.1 release: TraceQL performance improvements

Grafana Tempo 2.1 is out and comes with a host of TraceQL improvements. Tempo 2.1 comes with some nice incremental improvements to TraceQL and likely some breaking changes. There’s a section down below about those, too.

We’re excited to be delivering a much shorter release cycle since we introduced Tempo 2.0, which was roughly eight weeks ago. We would like to continue this cadence for the next few releases. Until then, read on to get a quick overview of the latest in Tempo. If you’re looking for something more in-depth, don’t hesitate to jump into the changelog or our Grafana Tempo 2.1 release notes.

You can also check out our webinar “Getting started with tracing and Grafana Tempo,” which is available on demand.

New TraceQL features and updates

Get ready for some new additions to TraceQL, the query language designed for selecting traces in Tempo. First up is the addition of the kind intrinsic to the language. Now a query like this may help you find any endpoint that took over 1s to respond:

{ kind = server && duration > 1s }

Next, we’ve added three new aggregate functions (min, max, and sum). Perhaps we’d like to find a query that spends more than 1s in aggregate querying a database:

{ span.db.statement != "" } | sum(duration) > 1s

Finally, there is now the ability to do arbitrary arithmetic in your queries. This will allow more human readable queries like:

{ span.bytesProcessed > 10 * 1024 * 1024 }

Or if you want to calculate and filter on a ratio:

{ span.bytesProcessed / span.jobs > 10 }

Expect to see a continuous flow of new TraceQL features and improvements in each Tempo release. This release also includes some nice performance improvements and an experimental vParquet2 that will set us up for structural operators in TraceQL.

Bug fixes 

• 2139 Fix float/int comparisons in TraceQL. This fixes { 1 = 1.0 }
• 2152 Fix not closing WAL block file before attempting to delete the folder.
• 2219 Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like status
• 2254 Correctly return unique spans when &&ing and ||ing spansets.
• 2289 Support negative values on aggregate filters like count() > -1.
• 2304 Support float as duration like {duration > 1.5s}

Breaking changes

• 2159 As promised, support for search on v2 blocks was removed. This includes the removal of three related configuration options: overrides:   max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:
• 2204 All metrics that were prefixed with cortex_ have been changed to tempo_
• 2008 SLO metrics were added to allow Tempo operators to better service their users and the metric query_frontend_result_metrics_inspected_bytes was removed in favor of query_frontend_bytes_processed_per_second.

What’s next in Grafana Tempo?

If I said TraceQL, would you be surprised? We are committed to continuing to flesh out this language and improve performance. Expect every release to continue to extend the language and reduce latency.

We do, however, have some exciting upcoming announcements in Tempo not related to TraceQL: streaming search APIs, metrics from traces, and improvements to the metrics generator come to mind. But there’s likely even more I’m forgetting about. So stay tuned!

If you are interested in hearing more about Grafana Tempo news or search progress, please join us on the Grafana Labs Community Slack channel #tempo, post a question in our community forums, reach out on Twitter, or join our monthly community call. See you there!

And if you want to get even closer to where the magic happens, why not have a look at our open positions at Grafana Labs? 

The easiest way to get started with Grafana Tempo is with Grafana Cloud, and our free forever tier now includes 50GB of traces along with 50GB of logs and 10K series of metrics. You can sign up for free!