---
title: "Version 2.3 release notes | Grafana Tempo documentation"
description: "Release notes for Grafana Tempo 2.3"
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Version 2.3 release notes

The Tempo team is pleased to announce the release of Tempo 2.3.

This release gives you:

- New [structural operators](../../traceql/construct-traceql-queries/#experimental-structural) in TraceQL:
  
  - Ancestor (`<<`)
  - Parent (`<`)
  - Not-child (`!>`) (experimental)
  - Not-descendant (`!>>`) (experimental)
  - Not-ancestor (`!<<`) (experimental)
  - Not-parent (`!<`) (experimental)
  - Not-siblings (`!~`) (experimental)
- TraceQL support for [searching quoted attributes](../../traceql/construct-traceql-queries/#experimental-structural) (for example, `{ span."name with spaces" = "foo" }`).
- [Dedicated attribute columns](../../operations/dedicated_columns/) for improving query performance on your most frequently queried attributes

Tempo 2.3 introduces [vParquet3](../../configuration/parquet/), a Parquet version designed to be more compatible with other Parquet implementations, available as a production-ready option. This block format improves query performance relative to previous formats.

Read the [**Tempo 2.3 blog post**](/blog/2023/11/01/grafana-tempo-2.3-release-faster-trace-queries-traceql-upgrades/) for more examples and details about these improvements.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases).

## Features and enhancements

The most important features and enhancements in Tempo 2.3 are highlighted below.

### vParquet3 promoted to stable

[vParquet3](../../configuration/parquet/), an updated Parquet block format introduced in Tempo 2.2, has been promoted to stable and is production ready. We’re excited about vParquet3 relative to prior formats because of its support for dedicated attribute columns, which help speed up queries on your largest and most queried attributes. \[[PR 2649](https://github.com/grafana/tempo/pull/2649)]

To read more about the design of vParquet3, refer to [the design proposal](https://github.com/grafana/tempo/blob/main/docs/design-proposals/2023-05%20vParquet3.md). For general information, refer to [the Apache Parquet schema](../../operations/schema/).

#### Dedicated attribute columns

Dedicated attribute columns improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list.

Introduced with vParquet3, dedicated attribute columns are only available when using this storage format.

To learn more about this new feature, refer to the [Dedicated attribute columns](../../operations/dedicated_columns/) documentation.

### TraceQL

Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, see the [TraceQL documentation](../../traceql/). For information on planned future extensions to the TraceQL language, refer to [future work](../../traceql/architecture/).

We’ve made the following improvements to TraceQL:

- Added two structural operators, ancestor (`<<`) and parent (`<`) ([documentation](../../traceql/construct-traceql-queries/#experimental-structural)) \[[PR 2877](https://github.com/grafana/tempo/pull/2877)]
- Added experimental not structural operators: `!>>`, `!>`, `!<<`, `!<`, and `!~` ([documentation](../../traceql/construct-traceql-queries/#experimental-structural)) \[[PR 2993](https://github.com/grafana/tempo/pull/2993)]
- Added quoted attribute syntax so you can search for attributes with spaces and other special characters in their names ([documentation](../../traceql/construct-traceql-queries/#experimental-structural)) \[[PR 3004](https://github.com/grafana/tempo/pull/3004)]
- Improved the performance of TraceQL [`select()` queries](../../traceql/construct-traceql-queries/#select-spans). Metrics-summary now also correctly handles missing attributes. \[[PR 2765](https://github.com/grafana/tempo/pull/2765)]
- Added support for searching by OpenTelemetry’s [span status message](https://github.com/open-telemetry/opentelemetry-proto/blob/afcd2aa7f728216d5891ffc0d83f09f0278a6611/opentelemetry/proto/trace/v1/trace.proto#L260) using `statusMessage` intrinsic attribute ([documentation](../../traceql/construct-traceql-queries/#intrinsic-fields)) \[[PR 2848](https://github.com/grafana/tempo/pull/2848)]
- Fixed cases where an empty filter (`{}`) didn’t return expected results \[[PR 2498](https://github.com/grafana/tempo/issues/2498)]

### Metrics-generator

We’ve made the following improvements to metrics-generator:

- Added a scope query parameter to `/api/overrides` so users can choose between fetching the overrides stored by the API and the merged overrides (those actually used by Tempo) \[[PR 2915](https://github.com/grafana/tempo/pull/2915), [#3018](https://github.com/grafana/tempo/pull/3018)]
- Added `TempoUserConfigurableOverridesReloadFailing` alert \[[PR 2784](https://github.com/grafana/tempo/pull/2784)]
- Added a metrics-generator configuration option to enable/disable X-Scope-OrgID headers on remote write to better support single-tenant setups. \[[PR 2974](https://github.com/grafana/tempo/pull/2974)]
- Allowed metrics-generator ingestion slack to be configured on a per-tenant, rather than global, basis \[[PR 2589](https://github.com/grafana/tempo/pull/2589)]
- Added several metrics-generator fields to user-configurable overrides, including histogram buckets, collection-interval, and span metrics filter policies \[[PR 2906](https://github.com/grafana/tempo/pull/2906), [2711](https://github.com/grafana/tempo/pull/2711), [2928](https://github.com/grafana/tempo/pull/2928), [2899](https://github.com/grafana/tempo/pull/2899)]

## Upgrade considerations

When upgrading to Tempo 2.3, be aware of these considerations and breaking changes.

### Transition to vParquet 3

Although the vParquet3 format isn’t yet the default, it’s production ready and we highly recommend switching to it for improved query performance and [dedicated attribute columns](../../operations/dedicated_columns/).

Upgrading to Tempo 2.3 doesn’t modify the Parquet block format. You can use Tempo 2.3 with vParquet2 or vParquet3. vParquet2 remains the default backend for Tempo 2.3; vParquet3 is available as a stable option.

> Note
> 
> Tempo 2.2 can’t read data stored in vParquet3.

For information on upgrading, refer to [Change the block format to vParquet3](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/) upgrade documentation.

### Azure SDK v2

If you are using Azure storage, we recommend using the v2 SDK, [azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go). You can use the `use_v2_sdk` configure option for switching. For more information, refer to the [Storage block configuration example documentation](/docs/tempo/next/configuration/#storage-block-configuration-example). \[[PR 2952](https://github.com/grafana/tempo/issues/2952)]

### Produce debug metrics with the distributor

You can now enable a new configuration block for the distributor to produce debug metrics. These metrics can be particularly useful when tracking down a process that’s creating an exceptional amount of spans. \[[PR 3008](https://github.com/grafana/tempo/pull/3008)]

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
distributor:
  metric_received_spans:
    enabled: false
    root_only: false
```

### Changes to the Overrides module configuration

We’ve added a new `defaults` block to the overrides module for configuring global or per-tenant settings. The Overrides change to indented syntax. For more information, read the [Overrides configuration documentation](../../configuration/#overrides).

You can also use the Tempo CLI to migrate configurations. Refer to the [documentation](../../operations/tempo_cli/#migrate-overrides-config-command). \[[PR 2688](https://github.com/grafana/tempo/pull/2688)]

The old configuration block looked like this:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]
```

Here is an example of the new configuration block:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]
```

### Removed or renamed configuration parameters

|                                                               |                                                                                                                                 |
|---------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|
| **Parameter**                                                 | **Comments**                                                                                                                    |
| `distributor.log_received_traces`                             | Use the `distributor.log_received_spans` configuration block instead. \[PR [#3008](https://github.com/grafana/tempo/pull/3008)] |
| `tempo_query_frontend_queries_total{op="searchtags|metrics"}` | Removed deprecated frontend metrics configuration option                                                                        |

## Security fixes

The following vulnerabilities have been addressed:

- Updated Alpine image version to 3.18 to patch [CVE-2022-48174](https://nvd.nist.gov/vuln/detail/CVE-2022-48174) [PR 3046](https://github.com/grafana/tempo/pull/3046)
- Corrected a HTTP rapid reset issue vulnerability ([CVE-2023-39325](https://github.com/advisories/GHSA-4374-p667-p6c8)) [PR 3017](https://github.com/grafana/tempo/pull/3017)

## Bugfixes

For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases/tag/v2.3.1).

### 2.3.1

- Include statusMessage intrinsic attribute in tag search. [PR 3084](https://github.com/grafana/tempo/pull/3084)
- Fix compactor ignore configured S3 headers. [PR 3149](https://github.com/grafana/tempo/pull/3154)
- Read session token to s3 credentials. [PR 3144](https://github.com/grafana/tempo/pull/3144)

### 2.3

- Loaded defaults for the internal server [PR 3041](https://github.com/grafana/tempo/pull/3041)
- Fixed pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions [PR 3012](https://github.com/grafana/tempo/pull/3012)
- Fixed a panic in metrics-summary API PR [#2738](https://github.com/grafana/tempo/pull/2738)
- Fixed a rare deadlock when uploading blocks to Azure Blob Storage [PR 2129](https://github.com/grafana/tempo/issues/2129)
- Only search ingester blocks that fall within the request time range. [PR 2783](https://github.com/grafana/tempo/pull/2783)
- Aligned `tempo_query_frontend_queries_total` and `tempo_query_frontend_queries_within_slo_total`. [PR 2840](https://github.com/grafana/tempo/pull/2840) This query now correctly tells you `%age` of requests that are within SLO:
  
  traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```traceql
  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op)
  /
  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
  ```
- Fixed support for blob storage in Azure Stack Hub as a backend. [PR 2853](https://github.com/grafana/tempo/pull/2853)
- Respected spss on GRPC streaming. [PR 2971](https://github.com/grafana/tempo/pull/2840)
- Moved empty root span substitution from `querier` to `query-frontend`. [PR 2671](https://github.com/grafana/tempo/issues/2671)
- Ingester errors correctly propagate on the query path [PR 2935](https://github.com/grafana/tempo/issues/2935)
- Fixed an issue where the ingester didn’t stop a query after timeout [PR 3031](https://github.com/grafana/tempo/pull/3031)
- Reordered the S3 credential chain and upgraded `minio-go`. `native_aws_auth_enabled` is deprecated [PR 3006](https://github.com/grafana/tempo/pull/3006)