---
title: "Version 3.1 release notes | Grafana Tempo documentation"
description: "Release notes for Grafana Tempo 3.1"
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Version 3.1 release notes

The Tempo team is pleased to announce the release of Grafana Tempo 3.1.

This release gives you:

- [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community.
- [Redaction by query](#redaction-by-query): Redact traces that match a TraceQL query instead of enumerating IDs, so a sensitive-data incident doesn’t require hunting down every affected trace.
- [TraceQL metrics extrapolation and arithmetic](#traceql-metrics-extrapolation-and-arithmetic): Extrapolate counts from head-sampled traces so rates reflect actual traffic, and combine aggregations with arithmetic to calculate ratios like error rate in a single query.
- [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost.
- [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans, with an experimental default-on option.
- [Trace diff and trace summary (experimental)](#trace-diff-and-trace-summary-experimental): Compare two traces to see what changed between a baseline and a candidate run, available through the API, `tempo-cli`, and the MCP server.
- [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks remain readable, with no migration.

These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases).

## Kafka ingestion improvements

In microservices mode, Tempo uses Kafka as a durable write-ahead log: traces flow through Kafka so distributors and consumers scale independently, and writes survive a consumer restart. Until 3.1, the Tempo Kafka client only supported SASL `PLAIN` authentication; it couldn’t encrypt in transit, prefer a local rack, or choose a compression codec. If your cluster required TLS or a stronger SASL mechanism, you couldn’t use this path at all.

Tempo 3.1 adds SASL `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, and `AWS_MSK_IAM` authentication alongside the existing `PLAIN` support, plus TLS and mutual-TLS, so you can connect to encrypted Kafka clusters, including Amazon MSK. A `client_rack` option enables rack-aware fetching (KIP-392). Consumers fetch from a replica in the same availability zone instead of always crossing zones to the partition leader, which cuts cross-zone data transfer cost on high-throughput pipelines. An `ingest.kafka.producer_compression` option overrides the producer’s compression codec, which Azure Event Hubs requires because it only supports `gzip`. (PRs [#7586](https://github.com/grafana/tempo/pull/7586), [#7594](https://github.com/grafana/tempo/pull/7594), [#7691](https://github.com/grafana/tempo/pull/7691))

Community contributors independently unblocked each of those gaps: SASL and TLS from [@heytrav](https://github.com/heytrav), rack-aware fetching from [@AvivGuiser](https://github.com/AvivGuiser), and gzip compression for Azure Event Hubs from [@fleighton](https://github.com/fleighton). Thanks to all three for this work.

Refer to [Configure authentication and TLS](/docs/tempo/next/set-up-for-tracing/setup-tempo/configure-kafka/#configure-authentication-and-tls) for SASL and TLS, and [Ingest](/docs/tempo/next/configuration/#ingest) for `client_rack` and `producer_compression`.

## Redaction by query

Tempo 3.0 introduced [trace redaction](/docs/tempo/next/release-notes/v3-0/#trace-redaction), but you had to supply an explicit list of trace IDs. Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you can match traces by attributes rather than enumerating IDs. \[[PR 7663](https://github.com/grafana/tempo/pull/7663)]

You can also bound a redaction to a time window with `--start` and `--end`, so a large tenant can be redacted in slices. \[[PR 7702](https://github.com/grafana/tempo/pull/7702)]

> Caution
> 
> Don’t submit a windowed redaction until every scheduler and worker is on Tempo 3.1. An older worker ignores the window and redacts every query match in the block, with no error.

This release also includes several [backend scheduler redaction reliability](#backend-scheduler-redaction-reliability) fixes.

Refer to [Redact traces](/docs/tempo/next/operations/tempo_cli/#redact-traces) for query syntax, constraints, and windowed redaction.

## TraceQL metrics extrapolation and arithmetic

If you use head-based sampling to control costs, metrics derived from those traces reflect only the sampled fraction, not actual traffic. At 10% sampling, `{ } | rate()` reports approximately 50 spans/s when the unsampled rate is 500 spans/s.

Tempo 3.1 lets TraceQL metrics extrapolate counts from the sampling probability recorded in each span, so rates and counts reflect actual traffic without a separate metrics pipeline. Opt in per query with the experimental `with(extrapolate=true)` hint. Extrapolation requires vParquet4 or later blocks; queries against vParquet3 blocks ignore the hint. \[[PR 7452](https://github.com/grafana/tempo/pull/7452)]

TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`) so you can combine aggregations in a single query instead of stitching results client-side. (PRs [#6866](https://github.com/grafana/tempo/pull/6866), [#7199](https://github.com/grafana/tempo/pull/7199), [#7409](https://github.com/grafana/tempo/pull/7409))

This query estimates the fraction of spans with error status, adjusted for sampling probability:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
({status=error} | rate()) / ({} | rate()) with(extrapolate=true)
```

For supported functions, requirements, and syntax, refer to [TraceQL metrics functions](/docs/tempo/next/metrics-from-traces/metrics-queries/functions/), including the [arithmetic operators](/docs/tempo/next/metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the [extrapolation hint](/docs/tempo/next/metrics-from-traces/metrics-queries/functions/#extrapolation-from-ingest-time-sampling-withextrapolatetrue-experimental).

## Metrics-generator and service graph improvements

If you use the metrics-generator, you get RED metrics and a service map from traces you already ingest. The generator processes every span at ingest, so CPU, restart time, and dropped edges all affect the series you alert on.

Service graphs get a new opt-in `traces_service_graph_connection_info` presence gauge, powered by new `service-graphs-*` subprocessors, for detecting service topology under heavy sampling, visualized in a new `tempo-service-graph.json` dashboard. \[[PR 7202](https://github.com/grafana/tempo/pull/7202)] The presence gauge keeps observed connections visible when sampling makes request-rate metrics sparse. It uses the same edge-detection rules as the existing service graph metrics.

Refer to [Connection information metric](/docs/tempo/next/metrics-from-traces/service_graphs/#connection-information-metric) to enable the subprocessor and query the gauge.

Additional service graph improvements:

- Service graphs now recognize the `db.system.name` attribute (the OpenTelemetry v1.30.0 rename of `db.system`) for database node detection and virtual node naming. Both attributes remain supported, so database nodes still appear after you upgrade to OpenTelemetry SDKs that emit the renamed attribute. \[[PR 7697](https://github.com/grafana/tempo/pull/7697), [documentation](/docs/tempo/next/metrics-from-traces/service_graphs/#database-name-attributes)]
- Expired service-graph edges are now labeled with the unmatched span kind, so you can tell which side of a connection was missing instead of an edge disappearing with no explanation. \[[PR 7709](https://github.com/grafana/tempo/pull/7709), [documentation](/docs/tempo/next/troubleshooting/metrics-generator/#expired-edges)]

The span-metrics and service-graphs processors were both reworked to build series labels through the metrics-generator registry’s pooled, borrowed-label path, cutting per-span and per-edge CPU and allocations. At high ingest rates, that per-span cost is what makes the generator expensive to run. Metric names, labels, and values are unchanged. (PRs [#7584](https://github.com/grafana/tempo/pull/7584), [#7587](https://github.com/grafana/tempo/pull/7587)) As part of this work, native histograms no longer attach an exemplar with an empty trace ID for spans that have no trace ID, matching classic histogram behavior.

A new `skip_stale_backlog_on_startup` option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. \[[PR 7611](https://github.com/grafana/tempo/pull/7611), [documentation](/docs/tempo/next/configuration/#metrics-generator)] A restart or scale-up no longer spends time processing spans that would never become metrics. Stale per-partition ingest lag metrics are now pruned when a partition moves between consumers, fixing ever-growing `tempo_ingest_group_partition_lag` series after a partition handoff. \[[PR 7665](https://github.com/grafana/tempo/pull/7665)] That series is how you tell whether the generator is keeping up; after a handoff it used to keep growing, which made health dashboards hard to trust.

## Span pruning for trace-by-id v2

The trace-by-id v2 endpoint can now prune uninteresting spans from the response to reduce payload size. \[[PR 7566](https://github.com/grafana/tempo/pull/7566)]

An experimental `span_pruning_enabled_by_default` option turns pruning on by default for v2 requests that don’t explicitly set `span_pruning`, with detection of traces already pruned on the write path so they aren’t re-pruned. \[[PR 7628](https://github.com/grafana/tempo/pull/7628)] A per-tenant `span_pruning_enabled` override lets individual tenants opt in or out of the default-on behavior, but only takes effect when the cluster-wide `span_pruning_enabled` is also `true`. \[[PR 7693](https://github.com/grafana/tempo/pull/7693)]

To help evaluate the effect of pruning on query results, the TraceQL engine has a new span-watcher framework for collecting extra query metrics on demand. Enabled through the experimental per-tenant `span_pruning_awareness` override, it reports whether matched spans include span-pruning summary spans, for both search and metrics queries. \[[PR 7532](https://github.com/grafana/tempo/pull/7532)]

Refer to [Query V2](/docs/tempo/next/api_docs/#query-v2) for the request parameters and [Query-frontend](/docs/tempo/next/configuration/#query-frontend) for the cluster-wide and per-tenant settings.

## Traces diff and trace summary (experimental)

> Note
> 
> Traces diff and trace summary are [experimental features](/docs/release-life-cycle/). Engineering and on-call support is not available. Experimental features might change or be removed in future releases and are not recommended for use in production environments.

Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. \[[PR 7539](https://github.com/grafana/tempo/pull/7539), [PR 7523](https://github.com/grafana/tempo/pull/7523)]

The HTTP API defaults to `trace-patch-v0` (the full span-level patch). A `trace-summary-v0-composed` format returns a compact summary alongside the patch, capping the patch at 64 KiB; larger patches report that the patch was omitted rather than truncating it silently. \[[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn’t produce false positives; the output’s duration field is now named `duration_nanos` and reports raw nanosecond values. \[[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. \[[PR 7564](https://github.com/grafana/tempo/pull/7564)]

`tempo-cli` also gets trace diff support for local work: `tempo-cli experimental trace-diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. \[[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)]

The MCP server also exposes a `trace-diff` tool for complete-trace comparisons, with a compact composed summary by default and patches up to 64 KiB. \[[PR 7785](https://github.com/grafana/tempo/pull/7785)]

Refer to [Trace diff](/docs/tempo/next/api_docs/#trace-diff) for the HTTP API, [Compare traces](/docs/tempo/next/api_docs/mcp-server/#compare-traces) for the MCP tool, and [Experimental trace diff](/docs/tempo/next/operations/tempo_cli/#experimental-trace-diff) for the CLI.

## Features and enhancements

The most important remaining features and enhancements in Tempo 3.1 are highlighted below.

### TraceQL correctness

- Instant metrics queries now correctly reuse the per-block job results cache. \[[PR 7602](https://github.com/grafana/tempo/pull/7602)]
- The vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as `trace:rootService` and `span:childCount`, array operations, or metrics queries on event and link intrinsics. (PRs [#7508](https://github.com/grafana/tempo/pull/7508), [#7533](https://github.com/grafana/tempo/pull/7533))
- `max_metrics_duration` is now enforced against the user-provided range rather than the post-alignment range. \[[PR 7170](https://github.com/grafana/tempo/pull/7170)]

### Performance improvements

- The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo/next/release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with the `metrics_spanonly_fetch: false` override. You can also disable it per-query with `with(spanonly_fetch=false)`, but this hint requires [`unsafe_query_hints`](/docs/tempo/next/configuration/#overrides) to be enabled for the tenant. \[[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo/next/metrics-from-traces/metrics-queries/#faster-read-path)]
- Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group; `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204))
- The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo/next/configuration/#cache))

### Backend scheduler redaction reliability

This release closes several correctness and scalability gaps in the backend scheduler’s redaction pipeline:

- A completed redaction batch now enters a short quiescence period before removal, so a block compacted right after the last job finishes still gets covered by a rescan. \[[PR 7695](https://github.com/grafana/tempo/pull/7695)]
- Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path. \[[PR 7153](https://github.com/grafana/tempo/pull/7153)]
- A dry-run redaction no longer disables the tenant’s compaction and retention or arms a rescan; those now only happen for a redaction in apply mode. \[[PR 7700](https://github.com/grafana/tempo/pull/7700)]
- A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could block future submissions. \[[PR 7703](https://github.com/grafana/tempo/pull/7703)]
- Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log surface when a job’s target block is missing from the live blocklist. \[[PR 7358](https://github.com/grafana/tempo/pull/7358)]
- A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode. A new Redaction row on the Backend Work dashboard visualizes these. \[[PR 7699](https://github.com/grafana/tempo/pull/7699)]
- A new `tempo_backend_scheduler_jobs_pending` metric reports queue depth. \[[PR 7772](https://github.com/grafana/tempo/pull/7772), [documentation](/docs/tempo/next/reference-tempo-architecture/components/compaction/#key-metrics)]
- Job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992))

For the full redaction job lifecycle and metrics, refer to [Compaction](/docs/tempo/next/reference-tempo-architecture/components/compaction/).

### Query and TraceQL

- The trace-by-id v2 endpoint adds filtering support with `q` (TraceQL filter) and `keep_hierarchy` query parameters. `match_depth` and `ancestor_depth` control how many descendant and ancestor hops to keep. \[[PR 7483](https://github.com/grafana/tempo/pull/7483), [PR 7708](https://github.com/grafana/tempo/pull/7708), [documentation](/docs/tempo/next/api_docs/#query-v2)]
- New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes, a `tempo_querier_backend_processing_duration_seconds` histogram, and stats metrics on additional querier methods. Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [#7571](https://github.com/grafana/tempo/pull/7571), [#7568](https://github.com/grafana/tempo/pull/7568), [documentation](/docs/tempo/next/reference-tempo-architecture/components/querier/#key-metrics))
- An opt-in `engine_bytes_tracking` override reports span and attribute size through the TraceQL engine as `tempo_query_frontend_engine_bytes_total`, so you can identify which queries scan the most data and plan capacity accordingly. \[[PR 7689](https://github.com/grafana/tempo/pull/7689), [documentation](/docs/tempo/next/configuration/#overrides)]
- The Tempo MCP server adds a `docs-config` tool and refreshed TraceQL and metrics documentation resources. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo/next/api_docs/mcp-server/#available-tools))

### Configuration and build

- New block-size histograms `tempo_block_builder_flush_size_bytes` and `tempodb_compaction_output_block_size_bytes` record the size of flushed and compacted blocks, helping you spot oversized blocks and tune compaction settings. \[[PR 7773](https://github.com/grafana/tempo/pull/7773)]
- A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can’t produce records to Kafka. \[[PR 7148](https://github.com/grafana/tempo/pull/7148)]
- The distributor now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click “Forget” after a non-graceful Pod termination. \[[PR 7098](https://github.com/grafana/tempo/pull/7098), [documentation](/docs/tempo/next/operations/manage-advanced-systems/consistent_hash_ring/#distributor)]
- Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance, so you can verify image authenticity before deploying. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo/next/operations/verify-container-images/))
- KEDA-based autoscaling is now available for live-store and for the metrics-generator in Jsonnet. Set `autoscaling_prometheus_tenant` when the Prometheus source is a multi-tenant system such as Grafana Mimir. (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [#7376](https://github.com/grafana/tempo/pull/7376), [documentation](/docs/tempo/next/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling))
- New Grafana dashboards: `tempo-service-graph.json` visualizes service topology, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [#7710](https://github.com/grafana/tempo/pull/7710), [documentation](/docs/tempo/next/operations/monitor/#dashboards))
- Darwin release builds are re-enabled. \[[PR 7407](https://github.com/grafana/tempo/pull/7407)]

## Upgrade considerations

When [upgrading](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.1, be aware of these considerations and breaking changes. Refer to [Upgrade to Tempo 3.1](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/#upgrade-to-tempo-31) for full migration steps.

Expand table

| Condition                                                | Action                                                                                                              |
|----------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| You write vParquet3 blocks                               | Change block version before upgrading. Refer to [vParquet3 deprecation enforced](#vparquet3-deprecation-enforced).  |
| You use Redis cache                                      | Review renamed YAML keys and removed Sentinel support. Refer to [Redis client rewrite](#redis-client-rewrite).      |
| You depend on a fixed Trace-by-ID shard count            | Set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Default value changes](#default-value-changes). |
| You depend on gRPC streaming responses larger than 1 MiB | Set `max_grpc_streaming_packet_size` explicitly. Refer to [Default value changes](#default-value-changes).          |

### Default block format is now vParquet5

Tempo 3.1 writes new blocks in vParquet5. Existing vParquet4 blocks are still readable, and no data migration is required. To keep writing vParquet4, set the block version explicitly:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
storage:
  trace:
    block:
      version: vParquet4
```

Refer to [Default block format is now vParquet5](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) for upgrade steps and [Apache Parquet block format](/docs/tempo/next/configuration/parquet/) for format details. \[[PR 7775](https://github.com/grafana/tempo/pull/7775)]

### vParquet3 deprecation enforced

vParquet3 was deprecated in Tempo 2.10 and 3.0, but nothing in code prevented its use. Tempo 3.1 enforces the deprecation: Tempo refuses to start if configured to write vParquet3 blocks, and existing vParquet3 blocks are no longer compacted. Reads of existing vParquet3 blocks are unaffected.

If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading. \[[PR 7858](https://github.com/grafana/tempo/pull/7858)]

Refer to [vParquet3 deprecation enforced](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/#vparquet3-deprecation-enforced) for upgrade steps and [Apache Parquet block format](/docs/tempo/next/configuration/parquet/) for format details.

### Redis client rewrite

The experimental Redis cache client has been completely rewritten:

- Redis Cluster is now the default routing mode,
- Redis Sentinel support is removed,
- several YAML keys are renamed, and
- the TLS block is replaced with a dskit-style block that fails closed on invalid configuration.

If you don’t use the Redis cache, no action is needed. Refer to [Redis cache configuration changes](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). \[[PR 7337](https://github.com/grafana/tempo/pull/7337)]

### Default value changes

- Query sharding: Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. \[[PR 7105](https://github.com/grafana/tempo/pull/7105), [documentation](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count)]
- Metrics query validation: A metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. \[[PR 7602](https://github.com/grafana/tempo/pull/7602)]
- gRPC streaming packet size: The query frontend’s default `max_grpc_streaming_packet_size` drops from 2 MiB to 1 MiB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. \[[PR 7615](https://github.com/grafana/tempo/pull/7615), [documentation](/docs/tempo/next/troubleshooting/querying/response-too-large/)]
- Memcached idle connections: Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. \[[PR 7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults)]

### Build and platform changes

- Tempo 3.1 upgrades to Go 1.27.1. Refer to [Security fixes](#security-fixes) for the CVEs this addresses.

## Security fixes

- Updated Go to 1.27.1 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to address [CVE-2026-39822](https://nvd.nist.gov/vuln/detail/CVE-2026-39822), [CVE-2026-42504](https://nvd.nist.gov/vuln/detail/CVE-2026-42504), [CVE-2026-27145](https://nvd.nist.gov/vuln/detail/CVE-2026-27145), [CVE-2026-42505](https://nvd.nist.gov/vuln/detail/CVE-2026-42505), [CVE-2026-42507](https://nvd.nist.gov/vuln/detail/CVE-2026-42507), [CVE-2026-46600](https://nvd.nist.gov/vuln/detail/CVE-2026-46600), and [CVE-2026-56852](https://nvd.nist.gov/vuln/detail/CVE-2026-56852). \[[PR 7641](https://github.com/grafana/tempo/pull/7641)]
- Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. \[[GHSA-jw7f-3cgp-hgrr](https://github.com/advisories/GHSA-jw7f-3cgp-hgrr), [PR 7153](https://github.com/grafana/tempo/pull/7153)]

## Bug fixes

For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases).

- Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. \[[PR 7787](https://github.com/grafana/tempo/pull/7787)]
- Fix classic histogram `_sum` series (for example, `traces_spanmetrics_latency_sum`) under-reporting via `increase()`/`rate()` for series with high churn. The `_sum` series was missing the zero-sample seed that `_count` and `_bucket` already had. \[[PR 7811](https://github.com/grafana/tempo/pull/7811)]
- Write the live-store per-block query-range response cache atomically, and log and ignore cache read errors instead of failing the query. \[[PR 7155](https://github.com/grafana/tempo/pull/7155)]
- Fix a data race in the query frontend when dispatching request batches to queriers with `max_batch_size > 1`. \[[PR 7664](https://github.com/grafana/tempo/pull/7664)]
- Reject TraceQL queries larger than `max_query_expression_size_bytes` before parsing them. \[[PR 7245](https://github.com/grafana/tempo/pull/7245), [documentation](/docs/tempo/next/configuration/#cap-the-maximum-query-length)]
- Fix `=nil` queries missing some values in TraceQL. \[[PR 7345](https://github.com/grafana/tempo/pull/7345)]
- Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, `{ resource.service.name = "foo" && name = }`). \[[PR 7660](https://github.com/grafana/tempo/pull/7660)]
- Propagate the `limit` and `maxStaleValues` tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. \[[PR 7609](https://github.com/grafana/tempo/pull/7609)]
- Fix a panic in TraceQL metrics-math queries (for example, `(A) / (B)`) when a sub-query’s `by()` clause uses the maximum number of group-by attributes. \[[PR 7831](https://github.com/grafana/tempo/pull/7831)]
- Cache empty tag-value results per block so a block with no matching values isn’t re-scanned on every query. \[[PR 7617](https://github.com/grafana/tempo/pull/7617)]
- Stop silently dropping a block’s tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. \[[PR 7610](https://github.com/grafana/tempo/pull/7610)]
- Fix a rare cache collision between instant and range metrics queries. \[[PR 7290](https://github.com/grafana/tempo/pull/7290)]
- Fix double path prefix in the compactor and `DeleteVersioned` on Azure and S3 backends. \[[PR 7271](https://github.com/grafana/tempo/pull/7271)]
- Preserve the no-compact flag when copying vParquet5 blocks so a freshly flushed block isn’t compacted or polled before it’s complete. \[[PR 7786](https://github.com/grafana/tempo/pull/7786)]
- Bound per-trace slice preallocation during distributor rebatching. \[[PR 7288](https://github.com/grafana/tempo/pull/7288)]
- Return a retryable status for transient errors writing to Kafka. \[[PR 7506](https://github.com/grafana/tempo/pull/7506)]
- Guard the `TempoDistributorKafkaProduceFailing` alert on a non-zero produce rate so it doesn’t page at `+Inf%` when no records have been produced yet. \[[PR 7715](https://github.com/grafana/tempo/pull/7715)]
- Fix per-tenant override for `retry_info_enabled` silently overriding the cluster default when unset. \[[PR 7662](https://github.com/grafana/tempo/pull/7662)]
- Fix the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`; it’s now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. \[[PR 7469](https://github.com/grafana/tempo/pull/7469)]
- Fix the packaged configuration for deb/rpm installs. Also adds CI validation and an errors-only flag for `-config.verify`. \[[PR 7830](https://github.com/grafana/tempo/pull/7830)]