This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Release notes V2.10

Version 2.10 release notes

The Tempo team is pleased to announce the release of Tempo 2.10.

This release gives you:

• New TraceQL features including minInt/maxInt constants and support for attribute = nil queries
• Enhanced metrics generator capabilities with entity-based limiting and overflow series
• Performance improvements and bug fixes across the platform
• Released vParquet5 encoding format as production-ready

These release notes highlight the most important features and bug fixes. For a complete list, refer to the Tempo CHANGELOG.

TraceQL

Tempo 2.10 introduces three TraceQL enhancements that help you query trace structure and data quality more precisely.

These features extend TraceQL’s ability to query both data completeness and trace structure—without requiring preprocessing or custom instrumentation. The = nil check and span:childCount intrinsic are useful for observability teams focused on instrumentation quality and performance optimization.

Find traces with missing attributes (= nil)

You can identify gaps in your instrumentation by finding spans where expected attributes are absent. For example, you can use nil to check for attributes that are missing or null and != nil to ensure an attribute is present with a non-null value. [PR 4905, documentation].

You can use this to:

• Auditing instrumentation coverage to find services that haven’t set required attributes like service.version or deployment.environment
• Detecting data quality issues to spot spans missing critical context before they cause blind spots in production

To audit instrumentation coverage, you could use the example query to find traces with missing attributes and identify the offending services:

traceql Copy

{ resource.service.version = nil || resource.deployment.environment = nil } | select(resource.service.name)

The = nil operator to matches spans where either service.version or deployment.environment is absent at the resource level. The select() operator returns the service name to identify which services need instrumentation updates.

The next example catches a common data quality issue: HTTP operations that were instrumented enough to capture the method but not the response status. This can create blind spots when analyzing error rates or latency by status code in production.

traceql Copy

{ span.http.method != nil && span.http.status_code = nil } | select(resource.service.name, span.http.route)

The != nil operator verifies the span has an HTTP method set. The = nil operator finds HTTP spans that lack a status code. The select() operator shows the service and route.

Analyze trace structure with span:childCount

Query spans based on how many direct child operations they trigger. [PR 6126, documentation]

This is useful for:

• Finding leaf spans (operations with no children), such as database queries or external API calls
• Detecting spans that spawn unusually many child operations, which can indicate N+1 query problems or inefficient batch processing

traceql Copy

{ span:childCount > 10 }

This query returns spans that trigger more than 10 child operations. A single request handler spawning dozens of database calls, for example, often points to an N+1 query problem where a loop issues one query per item instead of fetching all items in a single call.

Reference integer boundaries with minInt and maxInt

Use the named constants minInt and maxInt to represent the minimum and maximum integer values supported by TraceQL. [PR 5982]

This query finds spans where counter exists and is an integer.

traceql Copy

{ span.foo >= minInt }

Metrics-generator

Tempo 2.10 introduces improved cardinality management for the metrics-generator. Entity-based limiting is a new alternative to series-based limiting. It preserves complete metrics data for each logical entity rather than randomly dropping series when limits are reached. [PR 5788]

When limits are hit, the metrics-generator now produces overflow series labeled metric_overflow="true" instead of discarding data. This ensures no trace-derived metrics are lost. As stale series are pruned, new series automatically emerge from the overflow bucket. [PR 5954]

A new metric, tempo_metrics_generator_registry_active_series_demand_estimate, helps with capacity planning. It uses HyperLogLog estimation to report true cardinality demand even when limits cap the standard active series metric. The estimate has approximately 3% accuracy with minimal memory overhead. [PR 5710]

Additional metrics-generator improvements

This release also includes these metrics-generator updates:

• Drop and metric invalid UTF-8 for all metrics in metrics-generator instead of sending invalid data. [PR 5980]
• Add database_name_attributes configuration to the service graph processor to support the new db.namespace attribute. [PR 5398]
• Enhanced observability of collection failures in the metrics-generator with better error categorization. [PR 5936]
• Added validation for metrics-generator histogram buckets to prevent invalid configurations. [PR 5991]
• Added configuration to enable instance label for span metrics series, improving observability. [PR 5706]

vParquet5

vParquet5 is now production-ready. vParquet4 remains the default block format in this release.

vParquet5 improves query speed and reduces memory usage with these changes. Dedicated columns now support array values, enabling efficient storage for multi-valued attributes like HTTP headers. [PR 5760, documentation] Virtual span row numbers speed up queries that filter only on resource attributes. [PR 5943]

Dedicated columns now support event-level attributes. A new “blob” detection system identifies high-cardinality attributes like UUIDs and stack traces, then applies optimized encoding to reduce memory pressure and improve performance. The format automatically removes unused dedicated columns to save storage. [PR 5946, documentation]

Flexible dedicated column defaults replace the previous well-known attribute columns. [PR 5696, documentation]

The tempo-cli analyse block(s) command reports on integer attributes to help identify candidates for dedicated columns. [PR 6103, documentation]

vParquet5 also adds native storage for the span:childCount TraceQL intrinsic. [PR 6126, documentation]

Additional vParquet5 updates

This release also includes these vParquet5 improvements and fixes:

• Converted ServiceStats data from map to list and updated encoding and compression to enable sorted output and small reduction in file size. [PR 6216]
• Fixed issue with orphaned group nodes in vParquet5-preview6. [PR 6095]
• Fixed issues related to integer dedicated columns in vParquet5-preview2: [PR 5716]
• Fixed disappearing dedicated event attributes in trace view and dedicated blob column validation in vParquet5-preview6. [PR 6100]
• Fixed dedicated column array handling to better support attributes with mixed array and non-array values. [PR 6199]

LLM-optimized API responses

Tempo 2.10 introduces an LLM-optimized response format that improves how AI assistants process and understand trace data. When you send the Accept: application/vnd.grafana.llm header to supported endpoints, Tempo returns a simplified JSON structure that flattens the complex OpenTelemetry format into a more readable representation with computed values like duration in milliseconds. The Tempo MCP server (introduced in 2.9) uses this format automatically, so users querying traces through AI assistants like Claude Code or Cursor receive more accurate and useful responses without any configuration changes. This format is currently supported by the trace by ID and tag values endpoints. [PR 5962]

Features and enhancements

The most important features and enhancements in Tempo 2.10 are highlighted below.

TraceQL correctness

Tempo 2.10 adds new TraceQL functions, query capabilities, and related API improvements:

• Fixed incorrect results in TraceQL compare() caused by potential hash collision of string array attributes. [PR 5835]
• Fixed search by trace:id for short IDs with leading zeros. [PR 5587]
• Fixed rare bug where the TraceQL metrics could return an empty response when max series was hit. [PR 6099]
• Made top/bottomk deterministic by breaking ties with label values. [PR 5846]
• Fixed metrics streaming to release intermediate or incomplete results. [PR 5937]

Performance improvements

Tempo 2.10 includes these performance improvements:

• Enhanced TraceQL query performance through various optimizations. [PR 5773]
• Improved performance for quantile_over_time() function in TraceQL metrics queries. [PR 5996]
• Improved shutdown time in the first 30 seconds for faster graceful shutdowns. [PR 5725]
• Added metric for tracking added latency to write requests. [PR 5781]
• Improved error message when overrides fail to parse for better debugging. [PR 5787]

User-configurable overrides

The user-configurable overrides API now validates inputs more strictly (for example, rejecting duplicate labels) and exposes more metrics-generator settings for runtime configuration. This helps catch configuration mistakes earlier and fine-tune metrics generation per-tenant without restarts.

• Added stricter validation to user-configurable overrides API, such as refusing duplicate labels. [PR 6008, PR 6104]

• Exposed metrics-generator configuration options to the user-configurable overrides API, including dimension_mappings, intrinsic_dimensions, trace_id_label_name, and ingestion_time_range_slack. [PR 5989, PR 5974, PR 5972, PR 5958, PR 5973]

• Exposed metrics_generator.native_histogram_bucket_factor and metrics_generator.native_histogram_min_reset_duration to the user-configurable overrides API. [PR 5973]

• Added ability to disable RetryInfo in per-tenant overrides. [PR 5741]

Other enhancements

This release also includes these enhancements:

• Added support for application/protobuf content type in frontend endpoints for improved efficiency. [PR 5865]
• Added support for external storage to trace by id endpoint. [PR 6185]
• Added validation mode and tests for tempo-vulture. [PR 5605]
• Added SSE-C encryption support to S3 backend. [PR 5789]
• Added secure connection support to tempo-cli. [PR 5692]
• Added default_spans_per_span_set configuration option to control default spans returned per trace in search results. [PR 5858]
• Increased weight for heavy TraceQL queries. [PR 5782]
• Updated list of intrinsics returned by search tags endpoint. [PR 5857]
• Removed MustNewConstMetric to prevent panic and added validation for usage tracker configuration. Added tempo_distributor_usage_tracker_errors_total to surface errors in usage tracker. [PR 5981]
• Set maxKeys = 1 for Amazon S3 confirm list operation. [PR 6114]

Upgrade considerations

When upgrading to Tempo 2.10, be aware of these considerations and breaking changes.

Busybox removed from Tempo image

The Tempo container image no longer includes busybox. This change reduces the image size and attack surface, preventing future busybox-related vulnerabilities from affecting Tempo deployments. [PR 5717]

The image switched from gcr.io/distroless/static-debian12:debug, which includes busybox, to gcr.io/distroless/static-debian12, which doesn’t. The busybox shell and utilities are no longer available inside the running container.

You can no longer exec into the Tempo container with a shell. Commands like kubectl exec -it <pod> -- sh or docker exec -it <container> sh will fail.

To debug a running Tempo container, use one of these alternatives:

• Kubernetes ephemeral debug containers (kubectl debug)
• Docker Desktop or other container runtime tools that support shell injection for distroless images

If you have custom Docker Compose files or scripts that use the Tempo image for shell operations (such as running chown in an init container), update them to use a separate busybox:latest image for those tasks.

Tempo’s runtime behavior, configuration options, and APIs are unchanged.

vParquet format changes

This release includes breaking changes to vParquet block format support and deprecates older formats ahead of Tempo 3.0.

In preparation for Tempo 3.0, make sure you’re using vParquet4 or higher.

Enable vParquet5 (optional)

vParquet5 is now production-ready and available as an optional upgrade. vParquet4 remains the default block format. [PR 6219]

To enable vParquet5, set the block version in your storage configuration:

YAML Copy

storage:
  trace:
    block:
      version: vParquet5

Once enabled, Tempo writes new blocks in vParquet5 format while continuing to read existing vParquet4 blocks. No migration of existing data is required.

vParquet2 removed

vParquet2 encoding has been completely removed from Tempo 2.10. Tempo can no longer read vParquet2 blocks. [PR 6071]

No action is required if you’ve used default settings. The default block format migrated away from vParquet2 several releases ago.

Action is required if your storage configuration explicitly specifies vParquet2. Before upgrading, verify your configuration doesn’t specify vParquet2. If it does:

• Update to a previous Tempo release (2.9 or earlier) configured for vParquet3 or higher
• Wait for all existing vParquet2 blocks to expire and be deleted from backend storage
• Then upgrade to Tempo 2.10

Upgrading to 2.10 while vParquet2 blocks still exist in storage will cause read errors.

vParquet3 and vParquet2 schemas deprecated

TempoDB schemas vParquet3 and vParquet2 are now deprecated and will be removed in Tempo 3.0. [PR 6198]

If you’re using vParquet3, plan your migration to vParquet4 or higher before upgrading to Tempo 3.0.

Breaking changes

Be aware of these breaking changes when upgrading to Tempo 2.10:

• Tenant ID validation: Tempo now validates tenant IDs in the frontend and distributor components, using the same validation rules as Grafana Mimir. Previously, Tempo accepted any tenant ID value. Valid tenant IDs may only contain alphanumeric characters (a-z, A-Z, 0-9) and special characters (!, -, _, ., *, ', (, )), with a maximum length of 150 characters. Empty tenant IDs and the values . or .. are rejected. Requests with invalid tenant IDs return an error. [PR 5786]
• SearchTagsV2WithRange API change: The Tempo HTTP client SearchTagsV2WithRange function signature has changed. The function previously accepted only start and end parameters. It now requires two additional parameters: scope and query. This change affects Go code that imports the pkg/httpclient package. To maintain existing behavior, pass empty strings for the new parameters: SearchTagsV2WithRange("", "", start, end). [PR 6088]
• Go version upgrade: Tempo 2.10 upgrades to Go 1.25.5, which may affect custom builds or deployments with specific Go version requirements. (PRs #5939 #6001, #6096, #6089)

Project Rhythm: New Tempo architecture

Project Rhythm is the codename for the new architecture project for Tempo. The main objective of this project is to address certain trade-offs of the current design that are limiting the ability of Tempo to grow and support new functionalities.

The goals of this project are to:

• Eliminate the requirement for replication factor 3 (RF3) -> Support high availability and improve reliability of TraceQL metrics
• Decouple the read and write path -> Scalability and reliability
• Lay out the foundation to significantly reduce total cost of ownership (TCO) -> Room for growth

Project Rhythm has been discussed in the Tempo Community Calls for several months. Refer to the Tempo Community Calls on YouTube to learn more.

While the new architecture is available in Tempo, it is still considered experimental.

Tempo rearchitecture (experimental)

Refer to the Tempo rearchitecture section in the Tempo 2.10 CHANGELOG for the list of pull requests that are part of this project.

• Allow to configure WAL for livestore. [PR 6093]
• Added a single binary 3.0 mode --target=all-3.0 to begin testing single binary 3.0 and updating integration tests. This will be removed in 3.0 and become the standard single binary mode. [PR 6021]
• Kafka KIP-714 telemetry is enabled by default. Added disable_kafka_telemetry configuration flag to opt out. [PR 6046]
• Add query_end_cutoff setting to clamp near-“now” queries (default 30s) for consistent Tempo results. [PR 5682]
• Improve dashboards for livestore. [PR 5755]
• On startup, the first record for live store to consume is not older than two complete block timeouts. [PR 5693]
• Add endpoints for partition downscaling for live-store. [PR 5600, PR 5738, PR 5805, PR 5829]
• Add new livestore alert to the tempo-mixin. [PR 5752]
• Add partition ownership metric to live-store. [PR 5815]
• Simplify block-builder partition assignment with configuration parameter partitions_per_instance. [PR 6022]
• Fix block-builder to more precisely validate block encoding on startup. [PR 6037]
• Correctly track and reject too large traces in live stores. [PR 5757]
• Jsonnet: Correctly add tempo-gossip-member: true labels to block-builders and live-stores. [PR 6125]
• Fix wrong sleep duration in block-builder if one of partitions is inactive. [PR 5855]

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

• Prevent slice panic when truncating series after topk() by adding bounds check in metrics query-range combiner. [PR 6010]
• Fixed compactor to properly consider SSE-KMS information during metadata copy. [PR 5774]
• Fixed spss=0 parameter to properly mean unlimited spans instead of being rejected, and respect max_spans_per_span_set=0 configuration. [PR 5858]
• Fixed handle collisions with job and instance labels when targetInfo is enabled. [PR 5780]
• Fixed S3 compactor multipart upload to ensure Cloudflare R2 compliance by using uniform chunk sizes. [PR 5838]
• Fixed GetTrace() in tempo-query. [PR 5864]
• Fixed unsupported nonexistence nil operator for tags lookup. Fixed issue where some tag values/names were not returned due to distinctAttrCollector optimization. [PR 5967]
• Fixed delete implementation for S3, GCS, and Azure backends to account for prefix. [PR 6011]
• Fixed slow pod startup (~90s) in monolithic mode by returning false from isSharded() when kvstore is empty or in memory. [PR 6035]
• Fixed metrics generator to cache failed instances and back off when processor validation or instance creation fails, preventing indefinite retry attempts that could cause out-of-memory errors. [PR 6142]
• Fixed panic when trying to compact block with unsupported encodings such as vParquet previews. [PR 6209]
• Fixed leading zero handling in native histograms implementations. [PR 6033]