This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Release notes V3.1

Version 3.1 release notes

The Tempo team is pleased to announce the release of Grafana Tempo 3.1.

This release gives you:

• Kafka ingestion improvements: Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use gzip for Azure Event Hubs, all contributed by the Tempo community.
• Redaction by query: Redact traces that match a TraceQL query instead of enumerating IDs, so a sensitive-data incident doesn’t require hunting down every affected trace.
• TraceQL metrics extrapolation and arithmetic: Extrapolate counts from head-sampled traces so rates reflect actual traffic, and combine aggregations with arithmetic to calculate ratios like error rate in a single query.
• Metrics-generator and service graph improvements: Keep more service-map edges when you sample, recognize db.system.name, and cut per-span metrics-generator cost.
• Span pruning for trace-by-id v2: Trim response size by pruning uninteresting spans, with an experimental default-on option.
• Trace diff and trace summary (experimental): Compare two traces to see what changed between a baseline and a candidate run, available through the API, tempo-cli, and the MCP server.
• vParquet5 as the default block format: New blocks are written as vParquet5; existing vParquet4 blocks remain readable, with no migration.

These release notes highlight the most important features and bug fixes. For a complete list, refer to the Tempo CHANGELOG.

Kafka ingestion improvements

In microservices mode, Tempo uses Kafka as a durable write-ahead log: traces flow through Kafka so distributors and consumers scale independently, and writes survive a consumer restart. Until 3.1, the Tempo Kafka client only supported SASL PLAIN authentication; it couldn’t encrypt in transit, prefer a local rack, or choose a compression codec. If your cluster required TLS or a stronger SASL mechanism, you couldn’t use this path at all.

Tempo 3.1 adds SASL SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and AWS_MSK_IAM authentication alongside the existing PLAIN support, plus TLS and mutual-TLS, so you can connect to encrypted Kafka clusters, including Amazon MSK. A client_rack option enables rack-aware fetching (KIP-392). Consumers fetch from a replica in the same availability zone instead of always crossing zones to the partition leader, which cuts cross-zone data transfer cost on high-throughput pipelines. An ingest.kafka.producer_compression option overrides the producer’s compression codec, which Azure Event Hubs requires because it only supports gzip. (PRs #7586, #7594, #7691)

Community contributors independently unblocked each of those gaps: SASL and TLS from @heytrav, rack-aware fetching from @AvivGuiser, and gzip compression for Azure Event Hubs from @fleighton. Thanks to all three for this work.

Refer to Configure authentication and TLS for SASL and TLS, and Ingest for client_rack and producer_compression.

Redaction by query

Tempo 3.0 introduced trace redaction, but you had to supply an explicit list of trace IDs. Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you can match traces by attributes rather than enumerating IDs. [PR 7663]

You can also bound a redaction to a time window with --start and --end, so a large tenant can be redacted in slices. [PR 7702]

This release also includes several backend scheduler redaction reliability fixes.

Refer to Redact traces for query syntax, constraints, and windowed redaction.

TraceQL metrics extrapolation and arithmetic

If you use head-based sampling to control costs, metrics derived from those traces reflect only the sampled fraction, not actual traffic. At 10% sampling, { } | rate() reports approximately 50 spans/s when the unsampled rate is 500 spans/s.

Tempo 3.1 lets TraceQL metrics extrapolate counts from the sampling probability recorded in each span, so rates and counts reflect actual traffic without a separate metrics pipeline. Opt in per query with the experimental with(extrapolate=true) hint. Extrapolation requires vParquet4 or later blocks; queries against vParquet3 blocks ignore the hint. [PR 7452]

TraceQL metrics also adds arithmetic operators (+, -, *, /) so you can combine aggregations in a single query instead of stitching results client-side. (PRs #6866, #7199, #7409)

This query estimates the fraction of spans with error status, adjusted for sampling probability:

traceql Copy

({status=error} | rate()) / ({} | rate()) with(extrapolate=true)

For supported functions, requirements, and syntax, refer to TraceQL metrics functions, including the arithmetic operators and the extrapolation hint.

Metrics-generator and service graph improvements

If you use the metrics-generator, you get RED metrics and a service map from traces you already ingest. The generator processes every span at ingest, so CPU, restart time, and dropped edges all affect the series you alert on.

Service graphs get a new opt-in traces_service_graph_connection_info presence gauge, powered by new service-graphs-* subprocessors, for detecting service topology under heavy sampling, visualized in a new tempo-service-graph.json dashboard. [PR 7202] The presence gauge keeps observed connections visible when sampling makes request-rate metrics sparse. It uses the same edge-detection rules as the existing service graph metrics.

Refer to Connection information metric to enable the subprocessor and query the gauge.

Additional service graph improvements:

• Service graphs now recognize the db.system.name attribute (the OpenTelemetry v1.30.0 rename of db.system) for database node detection and virtual node naming. Both attributes remain supported, so database nodes still appear after you upgrade to OpenTelemetry SDKs that emit the renamed attribute. [PR 7697, documentation]
• Expired service-graph edges are now labeled with the unmatched span kind, so you can tell which side of a connection was missing instead of an edge disappearing with no explanation. [PR 7709, documentation]

The span-metrics and service-graphs processors were both reworked to build series labels through the metrics-generator registry’s pooled, borrowed-label path, cutting per-span and per-edge CPU and allocations. At high ingest rates, that per-span cost is what makes the generator expensive to run. Metric names, labels, and values are unchanged. (PRs #7584, #7587) As part of this work, native histograms no longer attach an exemplar with an empty trace ID for spans that have no trace ID, matching classic histogram behavior.

A new skip_stale_backlog_on_startup option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. [PR 7611, documentation] A restart or scale-up no longer spends time processing spans that would never become metrics. Stale per-partition ingest lag metrics are now pruned when a partition moves between consumers, fixing ever-growing tempo_ingest_group_partition_lag series after a partition handoff. [PR 7665] That series is how you tell whether the generator is keeping up; after a handoff it used to keep growing, which made health dashboards hard to trust.

Span pruning for trace-by-id v2

The trace-by-id v2 endpoint can now prune uninteresting spans from the response to reduce payload size. [PR 7566]

An experimental span_pruning_enabled_by_default option turns pruning on by default for v2 requests that don’t explicitly set span_pruning, with detection of traces already pruned on the write path so they aren’t re-pruned. [PR 7628] A per-tenant span_pruning_enabled override lets individual tenants opt in or out of the default-on behavior, but only takes effect when the cluster-wide span_pruning_enabled is also true. [PR 7693]

To help evaluate the effect of pruning on query results, the TraceQL engine has a new span-watcher framework for collecting extra query metrics on demand. Enabled through the experimental per-tenant span_pruning_awareness override, it reports whether matched spans include span-pruning summary spans, for both search and metrics queries. [PR 7532]

Refer to Query V2 for the request parameters and Query-frontend for the cluster-wide and per-tenant settings.

Traces diff and trace summary (experimental)

Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. [PR 7539, PR 7523]

The HTTP API defaults to trace-patch-v0 (the full span-level patch). A trace-summary-v0-composed format returns a compact summary alongside the patch, capping the patch at 64 KiB; larger patches report that the patch was omitted rather than truncating it silently. [PR 7593] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn’t produce false positives; the output’s duration field is now named duration_nanos and reports raw nanosecond values. [PR 7544] The combined size of both traces is checked against the max_bytes_per_trace per-tenant limit to protect the query frontend from oversized requests, returning 429 when exceeded. [PR 7564]

tempo-cli also gets trace diff support for local work: tempo-cli experimental trace-diff compares two local trace JSON files and emits trace-patch-v0 output, and an experimental trace-summary-v0-native format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [PR 7468, PR 7510]

The MCP server also exposes a trace-diff tool for complete-trace comparisons, with a compact composed summary by default and patches up to 64 KiB. [PR 7785]

Refer to Trace diff for the HTTP API, Compare traces for the MCP tool, and Experimental trace diff for the CLI.

Features and enhancements

The most important remaining features and enhancements in Tempo 3.1 are highlighted below.

TraceQL correctness

• Instant metrics queries now correctly reuse the per-block job results cache. [PR 7602]
• The vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as trace:rootService and span:childCount, array operations, or metrics queries on event and link intrinsics. (PRs #7508, #7533)
• max_metrics_duration is now enforced against the user-provided range rather than the post-alignment range. [PR 7170]

Performance improvements

• The faster span-only fetch path for metrics queries, introduced as experimental in Tempo 3.0, is now enabled by default. Disable it per-tenant with the metrics_spanonly_fetch: false override. You can also disable it per-query with with(spanonly_fetch=false), but this hint requires unsafe_query_hints to be enabled for the tenant. [PR 7179, documentation]
• Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group; ByteInPredicate/ByteNotInPredicate use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner. (PRs #7696, #7535, #7140, #7204)
• The Redis cache client supports a configurable max_item_size, and the Memcached client adds connect_timeout and min_idle_conns_headroom_percentage options. (PRs #7311, #7671, documentation)

Backend scheduler redaction reliability

This release closes several correctness and scalability gaps in the backend scheduler’s redaction pipeline:

• A completed redaction batch now enters a short quiescence period before removal, so a block compacted right after the last job finishes still gets covered by a rescan. [PR 7695]
• Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path. [PR 7153]
• A dry-run redaction no longer disables the tenant’s compaction and retention or arms a rescan; those now only happen for a redaction in apply mode. [PR 7700]
• A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could block future submissions. [PR 7703]
• Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log surface when a job’s target block is missing from the live blocklist. [PR 7358]
• A new tempo_backend_scheduler_redaction_traces_found_total metric reports how many traces a redaction job matched, split by apply and dry-run mode. A new Redaction row on the Backend Work dashboard visualizes these. [PR 7699]
• A new tempo_backend_scheduler_jobs_pending metric reports queue depth. [PR 7772, documentation]
• Job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs #7141, #6992)

For the full redaction job lifecycle and metrics, refer to Compaction.

Query and TraceQL

• The trace-by-id v2 endpoint adds filtering support with q (TraceQL filter) and keep_hierarchy query parameters. match_depth and ancestor_depth control how many descendant and ancestor hops to keep. [PR 7483, PR 7708, documentation]
• New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes, a tempo_querier_backend_processing_duration_seconds histogram, and stats metrics on additional querier methods. Query-shape span attribute names changed to snake_case (for example, queryType is now query_type). (PRs #7504, #7605, #7525, #7571, #7568, documentation)
• An opt-in engine_bytes_tracking override reports span and attribute size through the TraceQL engine as tempo_query_frontend_engine_bytes_total, so you can identify which queries scan the most data and plan capacity accordingly. [PR 7689, documentation]
• The Tempo MCP server adds a docs-config tool and refreshed TraceQL and metrics documentation resources. (PRs #7387, #7408, documentation)

Configuration and build

• New block-size histograms tempo_block_builder_flush_size_bytes and tempodb_compaction_output_block_size_bytes record the size of flushed and compacted blocks, helping you spot oversized blocks and tune compaction settings. [PR 7773]
• A new TempoDistributorKafkaProduceFailing alert fires when the distributor can’t produce records to Kafka. [PR 7148]
• The distributor now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click “Forget” after a non-graceful Pod termination. [PR 7098, documentation]
• Container image signing coverage is now complete: all four published images (tempo, tempo-vulture, tempo-query, tempo-cli) are signed with cosign and attested with SLSA build provenance, so you can verify image authenticity before deploying. (PRs #7601, #7543, #7493, documentation)
• KEDA-based autoscaling is now available for live-store and for the metrics-generator in Jsonnet. Set autoscaling_prometheus_tenant when the Prometheus source is a multi-tenant system such as Grafana Mimir. (PRs #7142, #7362, #7099, #7376, documentation)
• New Grafana dashboards: tempo-service-graph.json visualizes service topology, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. (PRs #7207, #7287, #7710, documentation)
• Darwin release builds are re-enabled. [PR 7407]

Upgrade considerations

When upgrading to Tempo 3.1, be aware of these considerations and breaking changes. Refer to Upgrade to Tempo 3.1 for full migration steps.

Default block format is now vParquet5

Tempo 3.1 writes new blocks in vParquet5. Existing vParquet4 blocks are still readable, and no data migration is required. To keep writing vParquet4, set the block version explicitly:

YAML Copy

storage:
  trace:
    block:
      version: vParquet4

Refer to Default block format is now vParquet5 for upgrade steps and Apache Parquet block format for format details. [PR 7775]

vParquet3 deprecation enforced

vParquet3 was deprecated in Tempo 2.10 and 3.0, but nothing in code prevented its use. Tempo 3.1 enforces the deprecation: Tempo refuses to start if configured to write vParquet3 blocks, and existing vParquet3 blocks are no longer compacted. Reads of existing vParquet3 blocks are unaffected.

If your storage configuration specifies vParquet3, change the block version to vParquet5 (default) or vParquet4 before upgrading. [PR 7858]

Refer to vParquet3 deprecation enforced for upgrade steps and Apache Parquet block format for format details.

Redis client rewrite

The experimental Redis cache client has been completely rewritten:

• Redis Cluster is now the default routing mode,
• Redis Sentinel support is removed,
• several YAML keys are renamed, and
• the TLS block is replaced with a dskit-style block that fails closed on invalid configuration.

If you don’t use the Redis cache, no action is needed. Refer to Redis cache configuration changes. [PR 7337]

Default value changes

• Query sharding: Trace-by-ID lookups now shard by block count through a new blocks_per_shard option, which defaults to 30 and takes precedence over the older query_shards setting. To keep the previous fixed-shard-count behavior, set blocks_per_shard: 0 to fall back to query_shards. [PR 7105, documentation]
• Metrics query validation: A metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [PR 7602]
• gRPC streaming packet size: The query frontend’s default max_grpc_streaming_packet_size drops from 2 MiB to 1 MiB. If you depend on larger streamed gRPC responses, set max_grpc_streaming_packet_size explicitly to restore the previous value. [PR 7615, documentation]
• Memcached idle connections: Idle Memcached connections are no longer closed after 2 minutes by default, and the default max_idle_conns is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set min_idle_conns_headroom_percentage to 0. [PR 7671, documentation]

Build and platform changes

• Tempo 3.1 upgrades to Go 1.27.1. Refer to Security fixes for the CVEs this addresses.

Security fixes

• Updated Go to 1.27.1 and bumped vendored golang.org/x/net and golang.org/x/text to address CVE-2026-39822, CVE-2026-42504, CVE-2026-27145, CVE-2026-42505, CVE-2026-42507, CVE-2026-46600, and CVE-2026-56852. [PR 7641]
• Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. [GHSA-jw7f-3cgp-hgrr, PR 7153]

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

• Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. [PR 7787]
• Fix classic histogram _sum series (for example, traces_spanmetrics_latency_sum) under-reporting via increase()/rate() for series with high churn. The _sum series was missing the zero-sample seed that _count and _bucket already had. [PR 7811]
• Write the live-store per-block query-range response cache atomically, and log and ignore cache read errors instead of failing the query. [PR 7155]
• Fix a data race in the query frontend when dispatching request batches to queriers with max_batch_size > 1. [PR 7664]
• Reject TraceQL queries larger than max_query_expression_size_bytes before parsing them. [PR 7245, documentation]
• Fix =nil queries missing some values in TraceQL. [PR 7345]
• Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, { resource.service.name = "foo" && name = }). [PR 7660]
• Propagate the limit and maxStaleValues tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. [PR 7609]
• Fix a panic in TraceQL metrics-math queries (for example, (A) / (B)) when a sub-query’s by() clause uses the maximum number of group-by attributes. [PR 7831]
• Cache empty tag-value results per block so a block with no matching values isn’t re-scanned on every query. [PR 7617]
• Stop silently dropping a block’s tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. [PR 7610]
• Fix a rare cache collision between instant and range metrics queries. [PR 7290]
• Fix double path prefix in the compactor and DeleteVersioned on Azure and S3 backends. [PR 7271]
• Preserve the no-compact flag when copying vParquet5 blocks so a freshly flushed block isn’t compacted or polled before it’s complete. [PR 7786]
• Bound per-trace slice preallocation during distributor rebatching. [PR 7288]
• Return a retryable status for transient errors writing to Kafka. [PR 7506]
• Guard the TempoDistributorKafkaProduceFailing alert on a non-zero produce rate so it doesn’t page at +Inf% when no records have been produced yet. [PR 7715]
• Fix per-tenant override for retry_info_enabled silently overriding the cluster default when unset. [PR 7662]
• Fix the version reported by --version, the build-info metric, and /api/status/buildinfo; it’s now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. [PR 7469]
• Fix the packaged configuration for deb/rpm installs. Also adds CI validation and an errors-only flag for -config.verify. [PR 7830]