Grafana Tempo 2.5 release: vParquet4, streaming endpoints, and more metrics

Grafana Tempo 2.5 is here with performance improvements, vParquet4 laying the groundwork for new TraceQL features, and additional metrics capabilities!

Watch the video above for a highlight of TraceQL metrics, or continue reading to get a quick overview of the latest updates in Tempo. If you’re looking for something more in-depth, don’t hesitate to jump into the Grafana Tempo 2.5 release notes or the changelog.

More TraceQL metrics

TraceQL metrics is an experimental feature in Grafana Tempo that dynamically creates metrics from traces. We’re working hard towards general availability, so stay tuned there.

In Tempo 2.4, we introduced the basic rate() method. Now, in 2.5, we are adding quantile_over_time() and histogram_over_time(), allowing users to aggregate numerical values, such as the all-important span duration. Notice that you can specify multiple quantiles in the same query.

{ name = "GET /:endpoint" } | quantile_over_time(duration, .99, .9, .5)

You can group by any span or resource attribute.

{ name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target)

Also, quantiles are not limited to just span duration; any numerical attribute on the span is fair game. To demonstrate this level of flexibility, consider this completely nonsensical quantile on span.http.status_code:

{ name = "GET /:endpoint" } | quantile_over_time(span.http.status_code, .99, .9, .5)

These features are cutting-edge and, as always, we recommend using the latest version of Grafana along with Tempo to have the best experience. 

More streaming endpoints

In 2.5, we went through a major refactor of the query frontend component in Tempo. This refactor included a number of performance improvements, but we primarily did it to add streaming gRPC endpoints across the board. 

Currently, Grafana only supports the TraceQL streaming search endpoint, but we’re looking to add support for others soon. I’m particularly excited about Grafana supporting the streaming metrics endpoint so we can see our pretty metrics lines drawn across the screen, as Tempo retrieves the data.

Even though Grafana does not yet support all of these endpoints, they can be called using an appropriately built gRPC client and are also supported by our Tempo CLI tool. For example, you could query streaming metrics like this:

./tempo-cli query api metrics localhost:3200 '{ status = error } | rate() by (name)' –use-grpc

vParquet4 paves the way for new features

vParquet4 is our latest Parquet version. It is available in Tempo 2.5, but will not be default until the next release. vParquet4 adds new columns to allow for querying on events, links, arrays, and more. Expect these features added to TraceQL soon!

To experiment with vParquet4 and be ready for the new features, you can configure Tempo to use it now. We’d like to thank Andreas Gerstmayr of Red Hat for adding columns and code to return span and error counts per service to our search results. We anticipate adding support in Grafana soon.

Important breaking changes

There are two breaking changes that are worth calling out in Tempo 2.5. 

First, the Tempo process in the official Docker image used to run as root. In this release, we have updated it to run as UID 10001 (a welcome security update). This means that components such as ingesters and metrics generators that maintain files on disk will not come up cleanly without intervention. The new user 10001 will not have access to the old files created by root. See the release notes for help on upgrading.

Next, the original vParquet format has been removed from Tempo 2.5. This means that direct upgrades from Tempo 2.1 to Tempo 2.5 are not possible. Operators will need to upgrade to an intermediate version and wait for the old vParquet blocks to fall out of retention before upgrading to 2.5.

For a full list of breaking changes, please check out the  Grafana Tempo 2.5 release notes or the changelog.

What’s next in Grafana Tempo?

There are two main initiatives in Tempo right now. The first is to GA TraceQL metrics and the second is to continue extending the TraceQL language.

In the coming releases, expect a steady stream of features detailed in the TraceQL language extensions now that we have vParquet4 officially in the codebase.

After a lot of hard work, we have determined that deduping RF3 traces for metrics is not going to hit the performance targets we would like. As a result, we are discussing architectural shifts in Tempo to allow for RF1 data to be written to object storage, while maintaining our current levels of availability and durability. If you’d like to participate in this conversation, please attend the monthly Tempo community call. We are tossing a lot of ideas around.

This will be a major shift in operating Tempo that we expect to result in lower TCO, faster TraceQL, and TraceQL metrics at scale. Expect big changes here!

If you are interested in hearing more about Grafana Tempo news or search progress, please join us on the Grafana Labs Community Slack channel #tempo, post a question in our community forums, reach out on X (formerly Twitter), or join our monthly Tempo community call. See you there!

And if you want to get even closer to where the magic happens, why not have a look at our open positions at Grafana Labs? 

The easiest way to get started with Grafana Tempo is with Grafana Cloud, and our free forever tier now includes 50GB of traces along with 50GB of logs and 10K series of metrics. You can sign up for free!