---
title: "Version 3.0 release notes | Grafana Tempo documentation"
description: "Release notes for Grafana Tempo 3.0"
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Version 3.0 release notes

The Tempo team is pleased to announce the release of Grafana Tempo 3.0.

This release gives you:

- [New architecture](#new-architecture): Splits the write and read path, offering increased reliability and scalability, lower TCO and moving TraceQL metrics to general availability.
- [TraceQL metrics generally available](#traceql-metrics): Production-ready trace-derived metrics at runtime with new comparison operators. Alerting on TraceQL metrics is an experimental capability.
- [Metrics-generator improvements](#metrics-generator): Per-label cardinality limiting, service graph filtering, and span multiplier support for sampled environments.
- [Trace redaction](#trace-redaction): New CLI command and API for hiding sensitive trace data.
- [Span profiling](#span-profiling): Native Pyroscope integration links profiling data to trace spans.
- [Query performance improvements](#query-performance-improvements): Doubled the number of dedicated columns and optimized TraceQL AST.

These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases).

## New architecture

Tempo 3.0 introduces a new architecture that replaces ingesters with a design that separates read and write paths. In microservices mode, a Kafka-compatible system provides durable buffering between ingestion and downstream components. In monolithic mode, Tempo runs all components in-process without Kafka.

Previously known as “Project Rhythm” (experimental in 2.9 and 2.10), this architecture is now the default and only ingest path.

For an overview of the components and data flow, refer to [Tempo architecture](/docs/tempo/next/introduction/architecture/).

### Greater resilience

In microservices mode, Kafka durably buffers trace data between ingestion and storage. Writes are acknowledged only after Kafka confirms receipt, ensuring durable ingestion even when downstream components are restarting or under load.

The read and write paths are fully decoupled and isolated from each other. Failures or load spikes on one path don’t affect the other.

Live-stores serve recent data from multiple availability zones, providing high availability without deduplication overhead.

### Lower total cost of ownership

The new architecture reduces costs in three ways.

First, it eliminates duplicated responsibilities where ingesters and the metrics-generator both created and flushed blocks.

Second, it reduces data in object storage. The previous architecture required replication factor 3 (RF3), which produced 2 to 2.5 times data duplication. The new design operates at RF1 with no duplication. Traces are sharded by ID to unique Kafka partitions, so each block is built exactly once.

Third, better autoscaling support lowers resource waste during low-traffic periods.

### Simpler operations

Each new component has a single, well-defined responsibility: block-builders create blocks, live-stores serve recent queries, and the backend scheduler and worker manage compaction and retention.

This clear separation makes the system easier to reason about and operate.

Compaction is now job-based, with centralized progress tracking, automatic rescheduling of failed jobs, and no duplicated work.

### What changed

- Block-builders consume from Kafka and build blocks for object storage (microservices mode).
- Live-stores serve recent data queries.
- The backend scheduler and worker replace the compactor for compaction and retention.

For full component details, refer to the [Tempo architecture](/docs/tempo/next/reference-tempo-architecture/) pages.

### Upgrade to 3.0

For migration steps, breaking changes, and upgrade considerations, refer to [Migrate to 3.0](/docs/tempo/next/set-up-for-tracing/setup-tempo/migrate-to-3/).

You can also use the [`tempo-cli migrate config`](/docs/tempo/next/operations/tempo_cli/#migrate-config-command) command to migrate your configuration from 2.x to 3.0. \[[PR 6982](https://github.com/grafana/tempo/pull/6982)]

## TraceQL metrics

TraceQL metrics move to general availability in Tempo 3.0. You can now build dashboards directly from trace data using metrics queries, without relying on the metrics-generator to pre-compute them. Refer to [TraceQL metrics queries](/docs/tempo/next/metrics-from-traces/metrics-queries/) to learn more.

> Note
> 
> While TraceQL metrics are generally available, alerting on TraceQL metrics is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available for the alerting use case.

### Comparison operators in metrics queries

With the new [comparison operators](/docs/tempo/next/metrics-from-traces/metrics-queries/functions/#comparison-operators), you can filter metrics results to only the data points that cross a threshold. Comparison operators are the foundation for alerting on trace-derived metrics: write a query that returns a value only when something is wrong, and point an alert rule at it. \[[PR 6474](https://github.com/grafana/tempo/pull/6474)]

For example, this query returns data only when a service exceeds 10 requests per second:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{} | rate() by (resource.service.name) > 10
```

You can also filter on latency. This query finds endpoints where the average span duration exceeds one second:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:name = "GET /:endpoint" } | avg_over_time(span:duration) > 1s
```

### Additional TraceQL improvements

- The default [`max_duration`](/docs/tempo/next/metrics-from-traces/metrics-queries/) for metrics queries increases to one day, so you can analyze longer time ranges without adjusting configuration. Default step intervals now align with vParquet5 timestamp columns for better query performance. \[[PR 6285](https://github.com/grafana/tempo/pull/6285), [PR 6413](https://github.com/grafana/tempo/pull/6413)]
- An experimental faster read path is available for most metrics queries behind the query hint `spanonly_fetch=true` when `unsafe_query_hints` is enabled. A per-tenant override controls opt-in or opt-out. (PRs [#6359](https://github.com/grafana/tempo/pull/6359), [#6849](https://github.com/grafana/tempo/pull/6849))
- Tag name and tag value autocomplete ( [search tags v2](/docs/tempo/next/api_docs/#search-tags-v2)) now supports `OR` conditions. When you search for attribute values in Grafana or through the API, Tempo can match against multiple values in a single request, returning results faster. \[[PR 6827](https://github.com/grafana/tempo/pull/6827)]

## Metrics-generator

Tempo 2.10 introduced entity-based limiting for the metrics-generator. Tempo 3.0 builds on this with per-label cardinality control and new filter policies for both span metrics and service graphs.

The new per-label limiter prevents a single high-cardinality label from exhausting your entire series budget. Set `max_cardinality_per_label` as a per-tenant override to cap how many distinct values any one label can have, while leaving other labels unaffected. Accompanying demand-estimate metrics help you tune limits before they kick in. \[[PR 6414](https://github.com/grafana/tempo/pull/6414), [documentation](/docs/tempo/next/reference-tempo-architecture/components/metrics-generator/)]

The span metrics filter picks up an `include_any` policy, making it easier to capture specific spans, such as internal traffic from a single service, without opening up your filter to all internal spans. \[[PR 6392](https://github.com/grafana/tempo/pull/6392), [documentation](/docs/tempo/next/metrics-from-traces/span-metrics/span-metrics-metrics-generator/#filtering)]

If you use head-based sampling, your metrics don’t reflect actual traffic volume. You can now set a `span_multiplier_key` per tenant to tell Tempo which span attribute holds the sampling ratio, so metric counts scale up to reflect your real request volume. \[[PR 6260](https://github.com/grafana/tempo/pull/6260), [documentation](/docs/tempo/next/metrics-from-traces/span-metrics/span-metrics-metrics-generator/)]

Service graphs now support the same `filter_policies` that span metrics already had. This means you can exclude noisy internal traffic or health checks from your service graph, keeping it clean and focused on the service-to-service communication you actually care about. \[[PR 6453](https://github.com/grafana/tempo/pull/6453), [documentation](/docs/tempo/next/metrics-from-traces/service_graphs/)]

### Additional metrics-generator improvements

- Support extracting span multiplier from W3C `tracestate` OpenTelemetry probability sampling threshold via the `enable_tracestate_span_multiplier` configuration option. \[[PR 6684](https://github.com/grafana/tempo/pull/6684)]
- Allow duplicate dimensions for span metrics and service graphs, supporting environments with different instrumentation libraries that use different attribute naming conventions. \[[PR 6288](https://github.com/grafana/tempo/pull/6288)]
- Allow `span_name_sanitization` to be set via the user-configurable overrides API. \[[PR 6411](https://github.com/grafana/tempo/pull/6411)]
- Deep validation for filter policies in the user-configurable overrides API. \[[PR 6407](https://github.com/grafana/tempo/pull/6407)]

## Trace redaction

Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire.

The [`tempo-cli redact`](/docs/tempo/next/operations/tempo_cli/#redact-traces) command submits redaction jobs to the [backend scheduler](/docs/tempo/next/reference-tempo-architecture/components/compaction/#backend-scheduler), which rewrites the affected blocks in object storage. You can monitor job progress through the [`/status/backendscheduler`](/docs/tempo/next/api_docs/#backend-scheduler-job-status) endpoint. \[[PR 6832](https://github.com/grafana/tempo/pull/6832)]

When individual span redaction isn’t enough, the `TraceRedactor` interface now supports hiding complete traces from query results through `ErrTraceHidden`. You can use this to enforce data governance policies that keep specific traces out of query results entirely. Refer to [Compaction](/docs/tempo/next/reference-tempo-architecture/components/compaction/#job-types) for details on how redaction jobs are scheduled and executed. \[[PR 6811](https://github.com/grafana/tempo/pull/6811)]

## Span profiling

Tempo 3.0 adds span profiling through [`otelpyroscope`](https://github.com/grafana/otel-profiling-go), so you can jump from a slow span in a trace directly to the CPU or memory profile that shows what that span was doing. Instead of correlating traces and profiles manually, you get a direct link between the two in Grafana Pyroscope, making it faster to find the root cause of latency or resource issues at the code level. \[[PR 7063](https://github.com/grafana/tempo/pull/7063)]

Under the hood, Tempo attaches pprof goroutine labels (`span_id`, `span_name`) to OpenTelemetry spans and adds a `pyroscope.profile.id` attribute to root spans.

Enable span profiling in your configuration file or with the CLI flag:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
span_profiling: true
```

Span profiling requires an OTLP exporter. Set `OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT`, or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` in your environment. Refer to [command-line flags](/docs/tempo/next/set-up-for-tracing/setup-tempo/command-line-flags/) for details.

## Query performance improvements

Tempo 3.0 speeds up queries through storage and query engine improvements. vParquet5 is now production-ready with expanded dedicated column support, and the TraceQL engine rewrites queries more efficiently.

### vParquet5

vParquet5 is now production-ready. vParquet4 remains the default block format in this release. vParquet5 includes all the capabilities introduced as opt-in in 2.10, including event-scoped columns, blob detection, and array-valued dedicated columns, plus expanded dedicated column support.

New in 3.0, the maximum number of dedicated string columns doubles to 20 per scope, so you can move even more of your most-searched attributes into fast dedicated storage. The `tempo-cli analyse blocks` command now recommends the optimal number for your data. \[[PR 6282](https://github.com/grafana/tempo/pull/6282)]

To enable vParquet5, set the block version in your storage configuration:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
storage:
  trace:
    block:
      version: vParquet5
```

After you enable vParquet5, Tempo writes new blocks in vParquet5 format while continuing to read existing vParquet4 blocks. No migration of existing data is required.

For details, refer to [Apache Parquet block format](/docs/tempo/next/configuration/parquet/) and [Dedicated attribute columns](/docs/tempo/next/operations/dedicated_columns/).

### TraceQL AST optimization

The TraceQL engine now automatically rewrites multiple conditions on the same attribute into a single array-aware check. This reduces the work the engine does per span and speeds up queries that filter on the same attribute more than once.

For example, this query evaluates more efficiently in 3.0 because the engine combines both conditions on `span.http.status_code` into one pass:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.status_code >= 500 && span.http.status_code < 600 }
```

This is a breaking change that alters how `!=` and `!~` behave on array attributes and introduces stricter validation of regular expression literals. Refer to [TraceQL array matching changes](#traceql-array-matching-changes) in the upgrade considerations for details on what to check before upgrading. \[[PR 6353](https://github.com/grafana/tempo/pull/6353)]

## Features and enhancements

The most important features and enhancements in Tempo 3.0 are highlighted below.

- Added `automemlimit` support for automatic GOMEMLIMIT configuration. Enable with `memory.automemlimit_enabled: true`. Refer to [Configuration](/docs/tempo/next/configuration/) for details. \[[PR 6313](https://github.com/grafana/tempo/pull/6313)]
- Added KEDA-based horizontal Pod autoscaling support for microservices deployment in Jsonnet. \[[PR 6970](https://github.com/grafana/tempo/pull/6970)]
- Enabled native histogram emission for all `promauto`-registered histograms, including `tempo_request_duration_seconds`. Both classic and native formats are emitted simultaneously; existing scrapers are unaffected. \[[PR 6910](https://github.com/grafana/tempo/pull/6910)]
- Added an extension mechanism for per-tenant overrides. Refer to [User-configurable overrides](/docs/tempo/next/operations/manage-advanced-systems/user-configurable-overrides/) for details. \[[PR 6758](https://github.com/grafana/tempo/pull/6758)]
- Added support for per-tenant left-padding of trace IDs. Refer to [Configuration](/docs/tempo/next/configuration/) for the `left_pad_trace_ids` parameter. \[[PR 6489](https://github.com/grafana/tempo/pull/6489)]
- Improved attribute truncating observability and logging of truncated oversized attributes. (PRs [#6400](https://github.com/grafana/tempo/pull/6400), [#6467](https://github.com/grafana/tempo/pull/6467))
- Exposed MinIO retry settings via S3 configuration. Refer to [Configuration](/docs/tempo/next/configuration/) for the `retry_max_attempts`, `retry_backoff_initial`, and `retry_backoff_max` parameters. \[[PR 6561](https://github.com/grafana/tempo/pull/6561)]
- Block builder now deduplicates spans within traces during block creation. Removed duplicates are tracked via the `tempo_block_builder_spans_deduped_total` metric. \[[PR 6539](https://github.com/grafana/tempo/pull/6539)]
- Added `--header` flag to `query api` commands for passing custom headers. Refer to the [Tempo CLI documentation](/docs/tempo/next/operations/tempo_cli/#query-api-command) for usage details. \[[PR 6768](https://github.com/grafana/tempo/pull/6768)]

## Upgrade considerations

When [upgrading](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.0, be aware of these considerations and breaking changes.

Tempo 3.0 is a major release with significant architectural changes. Use the `tempo-cli migrate config` command to automate configuration migration. Refer to the [Migrate from Tempo 2.x to 3.0](/docs/tempo/next/set-up-for-tracing/setup-tempo/migrate-to-3/) guide for step-by-step instructions.

### Ingester removal

The ingester module is removed entirely. All ingester-related configuration fields, CLI flags, alerts, and dashboard panels must be removed from your deployment. The write path is now exclusively handled by live-store and block-builder.

Removed configuration sections: `ingester`, `ingester_client`, `compactor`, `metrics_generator_client`. The `ingest.enabled` field is also removed, but the `ingest` block itself is still required for microservices mode (for example, `ingest.kafka`). (PRs [#6959](https://github.com/grafana/tempo/pull/6959), [#6504](https://github.com/grafana/tempo/pull/6504), [#6667](https://github.com/grafana/tempo/pull/6667), [#6873](https://github.com/grafana/tempo/pull/6873))

### Compactor removal

The compactor component and the `v2` block encoding are removed. Compaction is now handled by the [backend scheduler and worker](/docs/tempo/next/reference-tempo-architecture/components/compaction/), which track job progress centrally and automatically reschedule failed jobs.

Remove all compactor-related configuration, alerts, and dashboard panels from your deployment. The following CLI commands are also removed because they were specific to the `v2` format: `list block`, `list index`, `view index`, `gen index`, and `gen bloom`.

The compaction CLI flags drop their duplicate `compaction.` prefix. Update these flags in your configuration:

- `compaction.compaction.block-retention` → `compaction.block-retention`
- `compaction.compaction.max-objects-per-block` → `compaction.max-objects-per-block`
- `compaction.compaction.max-block-bytes` → `compaction.max-block-bytes`
- `compaction.compaction.compaction-window` → `compaction.compaction-window`

\[PRs [6273](https://github.com/grafana/tempo/pull/6273), [6369](https://github.com/grafana/tempo/pull/6369), [6909](https://github.com/grafana/tempo/pull/6909)]

### Target changes

The `all` target is now 3.0-compatible and the `scalable-single-binary` target is removed. Refer to [Deployment modes](/docs/tempo/next/reference-tempo-architecture/deployment-modes/) for details. \[[PR 6283](https://github.com/grafana/tempo/pull/6283)]

### Block and WAL configuration centralization

`block_builder` and `live_store` now always use `storage.trace.block` settings. Per-module block configuration fields are removed. If you configured block or WAL settings separately for these modules, move them to `storage.trace.block`. \[[PR 6647](https://github.com/grafana/tempo/pull/6647)]

### Other removed components

- The OpenCensus receiver is removed. Migrate to OTLP. \[[PR 6523](https://github.com/grafana/tempo/pull/6523)]
- The legacy `mem-ballast-size-mbs` CLI flag is removed. Use `GOMEMLIMIT` instead. \[[PR 6403](https://github.com/grafana/tempo/pull/6403)]
- `SpanMetricsSummary` is removed and querier code simplified. (PRs [#6496](https://github.com/grafana/tempo/pull/6496), [#6510](https://github.com/grafana/tempo/pull/6510))

### Configuration changes

- RetryInfo enabled by default: `distributor.retry_after_on_resource_exhausted` now defaults to `5s` (was `0`). OTLP clients receive a retry hint on `ResourceExhausted` errors. Set to `0` to disable cluster-wide, or set the per-tenant override `ingestion.retry_info_enabled: false` to disable for a single tenant. \[[PR 7088](https://github.com/grafana/tempo/pull/7088)]
- Read configuration consolidated: `query_frontend.search.query_ingesters_until` is removed in favor of `query_frontend.search.query_backend_after`. \[[PR 6507](https://github.com/grafana/tempo/pull/6507)]
- Legacy overrides disabled: Tempo refuses to start if legacy (flat, non-scoped) overrides are detected. Use `tempo-cli migrate overrides-config` to convert them. Set `enable_legacy_overrides: true` to opt back in temporarily. Refer to the [Migrate from Tempo 2.x to 3.0](/docs/tempo/next/set-up-for-tracing/setup-tempo/migrate-to-3/) guide for details. \[[PR 6741](https://github.com/grafana/tempo/pull/6741)]

### TraceQL array matching changes

The TraceQL AST optimization changes the semantics of `!=` and `!~` operators when used with array attributes. `!=` now means `NOT IN` (was `CONTAINS NOT EQUAL`) and `!~` now means `MATCH NONE` (was `CONTAINS NON-MATCH`). Regex operands must be of type string or string array. Disable the optimization with the query hint `skip_optimization=true` if needed. \[[PR 6353](https://github.com/grafana/tempo/pull/6353)]

### Tempo CLI timestamp format

The `query search` command no longer accepts timestamps without timezone (for example, `2024-01-01T00:00:00`). Use RFC3339 format (for example, `2024-01-01T00:00:00Z`) or relative time (for example, `now-1h`). Refer to the [Tempo CLI documentation](/docs/tempo/next/operations/tempo_cli/) for details. \[[PR 6458](https://github.com/grafana/tempo/pull/6458)]

### Go version upgrade

Tempo 3.0 upgrades to Go 1.26.2. \[[PR 6443](https://github.com/grafana/tempo/pull/6443)]

## Security fixes

- Fixed division by zero error in TraceQL expressions that could cause query failures. \[[PR 6580](https://github.com/grafana/tempo/pull/6580)]
- Fixed `intPow` function hanging for certain inputs, which could cause unbounded CPU consumption. \[[PR 6581](https://github.com/grafana/tempo/pull/6581)]
- Fixed integer overflow in query parameters by using `strconv.ParseUint` instead of `strconv.Atoi`/`strconv.ParseInt` for unsigned integer fields. \[[PR 6612](https://github.com/grafana/tempo/pull/6612)]
- Fixed allowlist header normalization when building the allowlist map. \[[PR 6481](https://github.com/grafana/tempo/pull/6481)]

## Bug fixes

For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases).

- Fixed incorrect search results for some queries on blob columns. \[[PR 6815](https://github.com/grafana/tempo/pull/6815)]
- Fixed active-series counter underflow in local series limiter when overflow series are deleted. \[[PR 6568](https://github.com/grafana/tempo/pull/6568)]
- Fixed per-label limiter and sanitizer being incorrectly applied to `target_info` and `host_info` metrics. \[[PR 6660](https://github.com/grafana/tempo/pull/6660)]
- Fixed `target_info` being skipped when resource attributes have empty values. \[[PR 6774](https://github.com/grafana/tempo/pull/6774)]
- Fixed limiter leak and permanent overflow when old series are replaced. \[[PR 6653](https://github.com/grafana/tempo/pull/6653)]
- Fixed dedicated column filtering issue. \[[PR 6586](https://github.com/grafana/tempo/pull/6586)]
- Fixed buffer-reuse bug where event attributes in dedicated columns could be persisted on additional spans and events. \[[PR 6914](https://github.com/grafana/tempo/pull/6914)]
- Fixed `span_name_sanitization` overrides not reloading during runtime. \[[PR 6435](https://github.com/grafana/tempo/pull/6435)]
- Returned 400 instead of 500 when `query_range` or `query_instant` requests have invalid start/end parameters. \[[PR 6694](https://github.com/grafana/tempo/pull/6694)]