This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Release notes V2.5

Version 2.5 release notes

The Tempo team is pleased to announce the release of Tempo 2.5.

This release gives you:

• New features, including more TraceQL metrics, gRPC streaming endpoints, and new experimental vParquet4 block format
• TraceQL enhancements and performance improvements
• Performance and stability enhancements

As part of this release, we’ve updated ownership for /var/tempo from root:root to a new tempo:tempo user with a UUID of 10001. Learn about this breaking change in Upgrade considerations.

Read the Tempo 2.5 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the Tempo changelog.

Features and enhancements

The most important features and enhancements in Tempo 2.5 are highlighted below.

Additional TraceQL metrics (experimental)

In this release, we’ve added several TraceQL metrics. Tempo 2.4 introduced the rate() function to view rates of spans. For this release, we’ve added quantile_over_time and histogram_over_time. PR 3605, PR 3633, PR 3644]

You can use quantiles_over_time allowing users to aggregate numerical values, such as the all-important span duration. Notice that you can specify multiple quantiles in the same query.

You can use historgram_over_time to display the metrics as a heatmap.

Note that using TraceQL metrics may require additional system resources.

For more information, refer to the TraceQL metrics queries and Configure TraceQL metrics.

gRPC streaming endpoints for Tempo APIs

With this feature, you can now see partial query results as they come in, so you no longer have to wait for the whole query to finish. This is perfect for big queries that take a long time to return a response.

The Tempo API endpoints now support gRPC streaming for tag queries and metrics. We’ve added new streaming endpoints for:

• SearchTags
• SearchTagsV2
• SearchTagValues
• SearchTagValuesV2
• MetricsQueryRange

These new streaming endpoints are not supported by Grafana yet. However, they are supported by the Tempo CLI tool.

In the Tempo CLI, you can use the --use-grpc option to enable GRPC streaming.

To learn more, refer to the Tempo gRPC API and Tempo CLI documentation. PR 3460 [PR #3584]

In addition, we’ve reduced memory consumption in the frontend for large traces. [PR 3522]

New vParquet4 block format (experimental)

New in Tempo 2.5, the vParquet4 block format is required for querying links, events, and arrays and improves query performance relative to previous formats. [PR 3368]

In addition, we’ve updated the OTLP schema to add attributes to instrumentation scope in vParquet4.[PR 3649]

While you can use vParquet4, keep in mind that it’s experimental. If you choose to use vParquet4 and then opt to revert to vParquet3, any vParquet4 blocks would not be readable by vParquet3.

To try vParquet4, refer to Choose a block format.

TraceQL

Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, see the TraceQL documentation.

For information on planned future extensions to the TraceQL language, refer to future work.

We’ve made the following improvements to TraceQL:

• Add support for scoped intrinsics using a colon (:). The available scoped intrinsics are trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage. [[PR 3629]
• Performance improvements on TraceQL and tag value search. [PR 3650,PR 3667]
• Add nestedSetLeft/Right/Parent intrinsics to TraceQL. [PR 3497]
• Reduce memory usage and improve performance when working with large datasets with many repeated strings. [PR 3411]
• Add support querying by trace:id and span:id [PR 3670]

Other enhancements and improvements

• Add TLS support for Memcached Client [PR 3585]
• Remove hardcoded delay in distributor shutdown [PR 3687]
• Surface new labels for uninstrumented services and systems [PR 3543]
• Add messaging-system latency histogram to service-graph [PR 3453]
• Add support for sharded ingester queries [PR 3574]
• Better compaction throughput and memory usage [PR 3579]
• Return a less confusing error message to the client when refusing spans due to ingestion rates. [PR 3485]
• Clean Metrics Generator’s Prometheus WAL before creating instance [PR 3548]
• Delete any remaining objects for empty tenants after a configurable duration, requires config enable PR 3611]

Upgrade considerations

When upgrading to Tempo 2.5, be aware of these considerations and breaking changes.

Docker image runs as new UID 10001

The Tempo process in the official Docker image used to run as root. The Tempo process now runs as UID 10001 in the Docker image.

Components such as ingesters and metrics generators that maintain files on disk won’t come up cleanly without intervention. The new user 10001 won’t have access to the old files created by root.

The ownership of /var/tempo changed from root:root to tempo:tempo with the UID/GID of 10001.

The ingester and metrics-generator statefulsets may need to run chown to change ownership to start properly.

Refer to PR 2265 to see a Jsonnet example of an init container.

This change doesn’t impact you if you used the Helm chart with the default security context set in the chart. All data should be owned by the tempo user already. The UID won’t impact Helm chart users.

Support for vParquet format removed

The original vParquet format has been removed from Tempo 2.5. Direct upgrades from Tempo 2.1 to Tempo 2.5 are not possible. You will need to upgrade to an intermediate version and wait for the old vParquet blocks to fall out of retention before upgrading to 2.5. PR 3663]

vParquet(1) won’t be recognized as a valid encoding and any remaining vParquet(1) blocks won’t be readable. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet(1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.

Updated, removed, or renamed configuration parameters

Additional considerations

• Updating to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers. PR 3649]
• Removes the addition of a tenant in multitenant trace id lookup. PR 3522]

Bugfixes

For a complete list, refer to the Tempo changelog.

• Fix handling of regex matchers in autocomplete endpoints PR 3641
• Return unfiltered results when a bad TraceQL query is provided in autocomplete. PR 3426
• Add support for dashes, quotes and spaces in attribute names in autocomplete PR 3458
• Fix metrics query results when filtering and rating on the same attribute PR 3428
• Fix metrics query results when series contain empty strings or nil values PR 3429
• Fix metrics query duration check, add per-tenant override for max metrics query duration PR 3479
• Fix metrics query panic “index out of range [-1]” when a trace has zero-length ID PR 3668
• Correctly handle 429s in GRPC search streaming. PR 3469
• Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. PR 3443
• Add spss and limit to the frontend cache key to prevent the return of incorrect results. PR 3557
• Use OS path separator to split blocks path. PR 3552
• Correctly parse TraceQL queries with > 1024 character attribute names or static values. PR 3571
• Fix span-metrics’ subprocessors bug that applied wrong configurations when running multiple tenants. PR 3612
• Fix panic in query-frontend when combining results PR 3683
• Fix TraceQL queries involving non boolean operations between statics and attributes. PR 3698