This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Release notes V3.0

Version 3.0 release notes

The Tempo team is pleased to announce the release of Grafana Tempo 3.0.

This release gives you:

• New architecture: Splits the write and read path, offering increased reliability and scalability, lower TCO and moving TraceQL metrics to general availability.
• TraceQL metrics generally available: Production-ready trace-derived metrics at runtime with new comparison operators. Alerting on TraceQL metrics is an experimental capability.
• Metrics-generator improvements: Per-label cardinality limiting, service graph filtering, and span multiplier support for sampled environments.
• Trace redaction: New CLI command and API for hiding sensitive trace data.
• Span profiling: Native Pyroscope integration links profiling data to trace spans.
• Query performance improvements: Doubled the number of dedicated columns and optimized TraceQL AST.

These release notes highlight the most important features and bug fixes. For a complete list, refer to the Tempo CHANGELOG.

New architecture

Tempo 3.0 introduces a new architecture that replaces ingesters with a design that separates read and write paths. In microservices mode, a Kafka-compatible system provides durable buffering between ingestion and downstream components. In monolithic mode, Tempo runs all components in-process without Kafka.

Previously known as “Project Rhythm” (experimental in 2.9 and 2.10), this architecture is now the default and only ingest path.

For an overview of the components and data flow, refer to Tempo architecture.

Greater resilience

In microservices mode, Kafka durably buffers trace data between ingestion and storage. Writes are acknowledged only after Kafka confirms receipt, ensuring durable ingestion even when downstream components are restarting or under load.

The read and write paths are fully decoupled and isolated from each other. Failures or load spikes on one path don’t affect the other.

Live-stores serve recent data from multiple availability zones, providing high availability without deduplication overhead.

Lower total cost of ownership

The new architecture reduces costs in three ways.

First, it eliminates duplicated responsibilities where ingesters and the metrics-generator both created and flushed blocks.

Second, it reduces data in object storage. The previous architecture required replication factor 3 (RF3), which produced 2 to 2.5 times data duplication. The new design operates at RF1 with no duplication. Traces are sharded by ID to unique Kafka partitions, so each block is built exactly once.

Third, better autoscaling support lowers resource waste during low-traffic periods.

Simpler operations

Each new component has a single, well-defined responsibility: block-builders create blocks, live-stores serve recent queries, and the backend scheduler and worker manage compaction and retention.

This clear separation makes the system easier to reason about and operate.

Compaction is now job-based, with centralized progress tracking, automatic rescheduling of failed jobs, and no duplicated work.

What changed

• Block-builders consume from Kafka and build blocks for object storage (microservices mode).
• Live-stores serve recent data queries.
• The backend scheduler and worker replace the compactor for compaction and retention.

For full component details, refer to the Tempo architecture pages.

Upgrade to 3.0

For migration steps, breaking changes, and upgrade considerations, refer to Migrate to 3.0.

You can also use the tempo-cli migrate config command to migrate your configuration from 2.x to 3.0. [PR 6982]

TraceQL metrics

TraceQL metrics move to general availability in Tempo 3.0. You can now build dashboards directly from trace data using metrics queries, without relying on the metrics-generator to pre-compute them. Refer to TraceQL metrics queries to learn more.

Comparison operators in metrics queries

With the new comparison operators, you can filter metrics results to only the data points that cross a threshold. Comparison operators are the foundation for alerting on trace-derived metrics: write a query that returns a value only when something is wrong, and point an alert rule at it. [PR 6474]

For example, this query returns data only when a service exceeds 10 requests per second:

traceql Copy

{} | rate() by (resource.service.name) > 10

You can also filter on latency. This query finds endpoints where the average span duration exceeds one second:

traceql Copy

{ span:name = "GET /:endpoint" } | avg_over_time(span:duration) > 1s

Additional TraceQL improvements

• The default max_duration for metrics queries increases to one day, so you can analyze longer time ranges without adjusting configuration. Default step intervals now align with vParquet5 timestamp columns for better query performance. [PR 6285, PR 6413]
• An experimental faster read path is available for most metrics queries behind the query hint spanonly_fetch=true when unsafe_query_hints is enabled. A per-tenant override controls opt-in or opt-out. (PRs #6359, #6849)
• Tag name and tag value autocomplete ( search tags v2) now supports OR conditions. When you search for attribute values in Grafana or through the API, Tempo can match against multiple values in a single request, returning results faster. [PR 6827]

Metrics-generator

Tempo 2.10 introduced entity-based limiting for the metrics-generator. Tempo 3.0 builds on this with per-label cardinality control and new filter policies for both span metrics and service graphs.

The new per-label limiter prevents a single high-cardinality label from exhausting your entire series budget. Set max_cardinality_per_label as a per-tenant override to cap how many distinct values any one label can have, while leaving other labels unaffected. Accompanying demand-estimate metrics help you tune limits before they kick in. [PR 6414, documentation]

The span metrics filter picks up an include_any policy, making it easier to capture specific spans, such as internal traffic from a single service, without opening up your filter to all internal spans. [PR 6392, documentation]

If you use head-based sampling, your metrics don’t reflect actual traffic volume. You can now set a span_multiplier_key per tenant to tell Tempo which span attribute holds the sampling ratio, so metric counts scale up to reflect your real request volume. [PR 6260, documentation]

Service graphs now support the same filter_policies that span metrics already had. This means you can exclude noisy internal traffic or health checks from your service graph, keeping it clean and focused on the service-to-service communication you actually care about. [PR 6453, documentation]

Additional metrics-generator improvements

• Support extracting span multiplier from W3C tracestate OpenTelemetry probability sampling threshold via the enable_tracestate_span_multiplier configuration option. [PR 6684]
• Allow duplicate dimensions for span metrics and service graphs, supporting environments with different instrumentation libraries that use different attribute naming conventions. [PR 6288]
• Allow span_name_sanitization to be set via the user-configurable overrides API. [PR 6411]
• Deep validation for filter policies in the user-configurable overrides API. [PR 6407]

Trace redaction

Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire.

The tempo-cli redact command submits redaction jobs to the backend scheduler, which rewrites the affected blocks in object storage. You can monitor job progress through the /status/backendscheduler endpoint. [PR 6832]

When individual span redaction isn’t enough, the TraceRedactor interface now supports hiding complete traces from query results through ErrTraceHidden. You can use this to enforce data governance policies that keep specific traces out of query results entirely. Refer to Compaction for details on how redaction jobs are scheduled and executed. [PR 6811]

Span profiling

Tempo 3.0 adds span profiling through otelpyroscope, so you can jump from a slow span in a trace directly to the CPU or memory profile that shows what that span was doing. Instead of correlating traces and profiles manually, you get a direct link between the two in Grafana Pyroscope, making it faster to find the root cause of latency or resource issues at the code level. [PR 7063]

Under the hood, Tempo attaches pprof goroutine labels (span_id, span_name) to OpenTelemetry spans and adds a pyroscope.profile.id attribute to root spans.

Enable span profiling in your configuration file or with the CLI flag:

YAML Copy

span_profiling: true

Span profiling requires an OTLP exporter. Set OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINT, or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT in your environment. Refer to command-line flags for details.

Query performance improvements

Tempo 3.0 speeds up queries through storage and query engine improvements. vParquet5 is now production-ready with expanded dedicated column support, and the TraceQL engine rewrites queries more efficiently.

vParquet5

vParquet5 is now production-ready. vParquet4 remains the default block format in this release. vParquet5 includes all the capabilities introduced as opt-in in 2.10, including event-scoped columns, blob detection, and array-valued dedicated columns, plus expanded dedicated column support.

New in 3.0, the maximum number of dedicated string columns doubles to 20 per scope, so you can move even more of your most-searched attributes into fast dedicated storage. The tempo-cli analyse blocks command now recommends the optimal number for your data. [PR 6282]

To enable vParquet5, set the block version in your storage configuration:

YAML Copy

storage:
  trace:
    block:
      version: vParquet5

After you enable vParquet5, Tempo writes new blocks in vParquet5 format while continuing to read existing vParquet4 blocks. No migration of existing data is required.

For details, refer to Apache Parquet block format and Dedicated attribute columns.

TraceQL AST optimization

The TraceQL engine now automatically rewrites multiple conditions on the same attribute into a single array-aware check. This reduces the work the engine does per span and speeds up queries that filter on the same attribute more than once.

For example, this query evaluates more efficiently in 3.0 because the engine combines both conditions on span.http.status_code into one pass:

traceql Copy

{ span.http.status_code >= 500 && span.http.status_code < 600 }

This is a breaking change that alters how != and !~ behave on array attributes and introduces stricter validation of regular expression literals. Refer to TraceQL array matching changes in the upgrade considerations for details on what to check before upgrading. [PR 6353]

Features and enhancements

The most important features and enhancements in Tempo 3.0 are highlighted below.

• Added automemlimit support for automatic GOMEMLIMIT configuration. Enable with memory.automemlimit_enabled: true. Refer to Configuration for details. [PR 6313]
• Added KEDA-based horizontal Pod autoscaling support for microservices deployment in Jsonnet. [PR 6970]
• Enabled native histogram emission for all promauto-registered histograms, including tempo_request_duration_seconds. Both classic and native formats are emitted simultaneously; existing scrapers are unaffected. [PR 6910]
• Added an extension mechanism for per-tenant overrides. Refer to User-configurable overrides for details. [PR 6758]
• Added support for per-tenant left-padding of trace IDs. Refer to Configuration for the left_pad_trace_ids parameter. [PR 6489]
• Improved attribute truncating observability and logging of truncated oversized attributes. (PRs #6400, #6467)
• Exposed MinIO retry settings via S3 configuration. Refer to Configuration for the retry_max_attempts, retry_backoff_initial, and retry_backoff_max parameters. [PR 6561]
• Block builder now deduplicates spans within traces during block creation. Removed duplicates are tracked via the tempo_block_builder_spans_deduped_total metric. [PR 6539]
• Added --header flag to query api commands for passing custom headers. Refer to the Tempo CLI documentation for usage details. [PR 6768]

Upgrade considerations

When upgrading to Tempo 3.0, be aware of these considerations and breaking changes.

Tempo 3.0 is a major release with significant architectural changes. Use the tempo-cli migrate config command to automate configuration migration. Refer to the Migrate from Tempo 2.x to 3.0 guide for step-by-step instructions.

Ingester removal

The ingester module is removed entirely. All ingester-related configuration fields, CLI flags, alerts, and dashboard panels must be removed from your deployment. The write path is now exclusively handled by live-store and block-builder.

Removed configuration sections: ingester, ingester_client, compactor, metrics_generator_client. The ingest.enabled field is also removed, but the ingest block itself is still required for microservices mode (for example, ingest.kafka). (PRs #6959, #6504, #6667, #6873)

Compactor removal

The compactor component and the v2 block encoding are removed. Compaction is now handled by the backend scheduler and worker, which track job progress centrally and automatically reschedule failed jobs.

Remove all compactor-related configuration, alerts, and dashboard panels from your deployment. The following CLI commands are also removed because they were specific to the v2 format: list block, list index, view index, gen index, and gen bloom.

The compaction CLI flags drop their duplicate compaction. prefix. Update these flags in your configuration:

• compaction.compaction.block-retention → compaction.block-retention
• compaction.compaction.max-objects-per-block → compaction.max-objects-per-block
• compaction.compaction.max-block-bytes → compaction.max-block-bytes
• compaction.compaction.compaction-window → compaction.compaction-window

[PRs 6273, 6369, 6909]

Target changes

The all target is now 3.0-compatible and the scalable-single-binary target is removed. Refer to Deployment modes for details. [PR 6283]

Block and WAL configuration centralization

block_builder and live_store now always use storage.trace.block settings. Per-module block configuration fields are removed. If you configured block or WAL settings separately for these modules, move them to storage.trace.block. [PR 6647]

Other removed components

• The OpenCensus receiver is removed. Migrate to OTLP. [PR 6523]
• The legacy mem-ballast-size-mbs CLI flag is removed. Use GOMEMLIMIT instead. [PR 6403]
• SpanMetricsSummary is removed and querier code simplified. (PRs #6496, #6510)

Configuration changes

• RetryInfo enabled by default: distributor.retry_after_on_resource_exhausted now defaults to 5s (was 0). OTLP clients receive a retry hint on ResourceExhausted errors. Set to 0 to disable cluster-wide, or set the per-tenant override ingestion.retry_info_enabled: false to disable for a single tenant. [PR 7088]
• Read configuration consolidated: query_frontend.search.query_ingesters_until is removed in favor of query_frontend.search.query_backend_after. [PR 6507]
• Legacy overrides disabled: Tempo refuses to start if legacy (flat, non-scoped) overrides are detected. Use tempo-cli migrate overrides-config to convert them. Set enable_legacy_overrides: true to opt back in temporarily. Refer to the Migrate from Tempo 2.x to 3.0 guide for details. [PR 6741]

TraceQL array matching changes

The TraceQL AST optimization changes the semantics of != and !~ operators when used with array attributes. != now means NOT IN (was CONTAINS NOT EQUAL) and !~ now means MATCH NONE (was CONTAINS NON-MATCH). Regex operands must be of type string or string array. Disable the optimization with the query hint skip_optimization=true if needed. [PR 6353]

Tempo CLI timestamp format

The query search command no longer accepts timestamps without timezone (for example, 2024-01-01T00:00:00). Use RFC3339 format (for example, 2024-01-01T00:00:00Z) or relative time (for example, now-1h). Refer to the Tempo CLI documentation for details. [PR 6458]

Go version upgrade

Tempo 3.0 upgrades to Go 1.26.2. [PR 6443]

Security fixes

• Fixed division by zero error in TraceQL expressions that could cause query failures. [PR 6580]
• Fixed intPow function hanging for certain inputs, which could cause unbounded CPU consumption. [PR 6581]
• Fixed integer overflow in query parameters by using strconv.ParseUint instead of strconv.Atoi/strconv.ParseInt for unsigned integer fields. [PR 6612]
• Fixed allowlist header normalization when building the allowlist map. [PR 6481]

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

• Fixed incorrect search results for some queries on blob columns. [PR 6815]
• Fixed active-series counter underflow in local series limiter when overflow series are deleted. [PR 6568]
• Fixed per-label limiter and sanitizer being incorrectly applied to target_info and host_info metrics. [PR 6660]
• Fixed target_info being skipped when resource attributes have empty values. [PR 6774]
• Fixed limiter leak and permanent overflow when old series are replaced. [PR 6653]
• Fixed dedicated column filtering issue. [PR 6586]
• Fixed buffer-reuse bug where event attributes in dedicated columns could be persisted on additional spans and events. [PR 6914]
• Fixed span_name_sanitization overrides not reloading during runtime. [PR 6435]
• Returned 400 instead of 500 when query_range or query_instant requests have invalid start/end parameters. [PR 6694]