This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Release notes V2.3

Version 2.3 release notes

The Tempo team is pleased to announce the release of Tempo 2.3.

This release gives you:

• New structural operators in TraceQL:

  • Ancestor (<<)
  • Parent (<)
  • Not-child (!>) (experimental)
  • Not-descendant (!>>) (experimental)
  • Not-ancestor (!<<) (experimental)
  • Not-parent (!<) (experimental)
  • Not-siblings (!~) (experimental)

• TraceQL support for searching quoted attributes (for example, { span."name with spaces" = "foo" }).

• Dedicated attribute columns for improving query performance on your most frequently queried attributes

Tempo 2.3 introduces vParquet3, a Parquet version designed to be more compatible with other Parquet implementations, available as a production-ready option. This block format improves query performance relative to previous formats.

Read the Tempo 2.3 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the Tempo changelog.

Features and enhancements

The most important features and enhancements in Tempo 2.3 are highlighted below.

vParquet3 promoted to stable

vParquet3, an updated Parquet block format introduced in Tempo 2.2, has been promoted to stable and is production ready. We’re excited about vParquet3 relative to prior formats because of its support for dedicated attribute columns, which help speed up queries on your largest and most queried attributes. [PR 2649]

To read more about the design of vParquet3, refer to the design proposal. For general information, refer to the Apache Parquet schema.

Dedicated attribute columns

Dedicated attribute columns improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list.

Introduced with vParquet3, dedicated attribute columns are only available when using this storage format.

To learn more about this new feature, refer to the Dedicated attribute columns documentation.

TraceQL

Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, refer to future work.

We’ve made the following improvements to TraceQL:

• Added two structural operators, ancestor (<<) and parent (<) (documentation) [PR 2877]

• Added experimental not structural operators: !>>, !>, !<<, !<, and !~ (documentation) [PR 2993]

• Added quoted attribute syntax so you can search for attributes with spaces and other special characters in their names (documentation) [PR 3004]

• Improved the performance of TraceQL select() queries. Metrics-summary now also correctly handles missing attributes. [PR 2765]

• Added support for searching by OpenTelemetry’s span status message using statusMessage intrinsic attribute (documentation) [PR 2848]

• Fixed cases where an empty filter ({}) didn’t return expected results [PR 2498]

Metrics-generator

We’ve made the following improvements to metrics-generator:

• Added a scope query parameter to /api/overrides so users can choose between fetching the overrides stored by the API and the merged overrides (those actually used by Tempo) [PR 2915, #3018]
• Added TempoUserConfigurableOverridesReloadFailing alert [PR 2784]
• Added a metrics-generator configuration option to enable/disable X-Scope-OrgID headers on remote write to better support single-tenant setups. [PR 2974]
• Allowed metrics-generator ingestion slack to be configured on a per-tenant, rather than global, basis [PR 2589]
• Added several metrics-generator fields to user-configurable overrides, including histogram buckets, collection-interval, and span metrics filter policies [PR 2906, 2711, 2928, 2899]

Upgrade considerations

When upgrading to Tempo 2.3, be aware of these considerations and breaking changes.

Transition to vParquet 3

Although the vParquet3 format isn’t yet the default, it’s production ready and we highly recommend switching to it for improved query performance and dedicated attribute columns.

Upgrading to Tempo 2.3 doesn’t modify the Parquet block format. You can use Tempo 2.3 with vParquet2 or vParquet3. vParquet2 remains the default backend for Tempo 2.3; vParquet3 is available as a stable option.

For information on upgrading, refer to Change the block format to vParquet3 upgrade documentation.

Azure SDK v2

If you are using Azure storage, we recommend using the v2 SDK, azure-sdk-for-go. You can use the use_v2_sdk configure option for switching. For more information, refer to the Storage block configuration example documentation. [PR 2952]

Produce debug metrics with the distributor

You can now enable a new configuration block for the distributor to produce debug metrics. These metrics can be particularly useful when tracking down a process that’s creating an exceptional amount of spans. [PR 3008]

yaml Copy

distributor:
  metric_received_spans:
    enabled: false
    root_only: false

Changes to the Overrides module configuration

We’ve added a new defaults block to the overrides module for configuring global or per-tenant settings. The Overrides change to indented syntax. For more information, read the Overrides configuration documentation.

You can also use the Tempo CLI to migrate configurations. Refer to the documentation. [PR 2688]

The old configuration block looked like this:

yaml Copy

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

Here is an example of the new configuration block:

yaml Copy

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Removed or renamed configuration parameters

Security fixes

The following vulnerabilities have been addressed:

• Updated Alpine image version to 3.18 to patch CVE-2022-48174 PR 3046
• Corrected a HTTP rapid reset issue vulnerability (CVE-2023-39325) PR 3017

Bugfixes

For a complete list, refer to the Tempo changelog.

2.3.1

• Include statusMessage intrinsic attribute in tag search. PR 3084
• Fix compactor ignore configured S3 headers. PR 3149
• Readd session token to s3 credentials. PR 3144

2.3

• Loaded defaults for the internal server PR 3041

• Fixed pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions PR 3012

• Fixed a panic in metrics-summary API PR #2738

• Fixed a rare deadlock when uploading blocks to Azure Blob Storage PR 2129

• Only search ingester blocks that fall within the request time range. PR 2783

• Aligned tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. PR 2840 This query now correctly tells you %age of requests that are within SLO:

  Copy

  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op)
  /
  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)

• Fixed support for blob storage in Azure Stack Hub as a backend. PR 2853

• Respected spss on GRPC streaming. PR 2971

• Moved empty root span substitution from querier to query-frontend. PR 2671

• Ingester errors correctly propagate on the query path PR 2935

• Fixed an issue where the ingester didn’t stop a query after timeout PR 3031

• Reordered the S3 credential chain and upgraded minio-go. native_aws_auth_enabled is deprecated PR 3006