Release notes on Grafana Labs

Version 2.10 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.10 release notes

The Tempo team is pleased to announce the release of Tempo 2.10.

This release gives you:

New TraceQL features including minInt/maxInt constants and support for attribute = nil queries
Enhanced metrics generator capabilities with entity-based limiting and overflow series
Performance improvements and bug fixes across the platform
Released vParquet5 encoding format as production-ready

These release notes highlight the most important features and bug fixes. For a complete list, refer to the Tempo CHANGELOG.

To learn more about this release, refer to the Tempo 2.10 blog post.

TraceQL

Tempo 2.10 introduces three TraceQL enhancements that help you query trace structure and data quality more precisely.

These features extend TraceQL’s ability to query both data completeness and trace structure—without requiring preprocessing or custom instrumentation. The = nil check and span:childCount intrinsic are useful for observability teams focused on instrumentation quality and performance optimization.

Find traces with missing attributes (`= nil`)

You can identify gaps in your instrumentation by finding spans where expected attributes are absent. For example, you can use nil to check for attributes that are missing or null and != nil to ensure an attribute is present with a non-null value. [PR 4905, documentation].

You can use this to:

Auditing instrumentation coverage to find services that haven’t set required attributes like service.version or deployment.environment
Detecting data quality issues to spot spans missing critical context before they cause blind spots in production

To audit instrumentation coverage, you could use the example query to find traces with missing attributes and identify the offending services:

{ resource.service.version = nil || resource.deployment.environment = nil } | select(resource.service.name)

The = nil operator to matches spans where either service.version or deployment.environment is absent at the resource level. The select() operator returns the service name to identify which services need instrumentation updates.

The next example catches a common data quality issue: HTTP operations that were instrumented enough to capture the method but not the response status. This can create blind spots when analyzing error rates or latency by status code in production.

{ span.http.method != nil && span.http.status_code = nil } | select(resource.service.name, span.http.route)

The != nil operator verifies the span has an HTTP method set. The = nil operator finds HTTP spans that lack a status code. The select() operator shows the service and route.

Analyze trace structure with `span:childCount`

Query spans based on how many direct child operations they trigger. [PR 6126, documentation]

This is useful for:

Finding leaf spans (operations with no children), such as database queries or external API calls
Detecting spans that spawn unusually many child operations, which can indicate N+1 query problems or inefficient batch processing

{ span:childCount > 10 }

This query returns spans that trigger more than 10 child operations. A single request handler spawning dozens of database calls, for example, often points to an N+1 query problem where a loop issues one query per item instead of fetching all items in a single call.

Reference integer boundaries with `minInt` and `maxInt`

Use the named constants minInt and maxInt to represent the minimum and maximum integer values supported by TraceQL. [PR 5982]

This query finds spans where counter exists and is an integer.

{ span.foo >= minInt }

Metrics-generator

Tempo 2.10 introduces improved cardinality management for the metrics-generator. Entity-based limiting is a new alternative to series-based limiting. It preserves complete metrics data for each logical entity rather than randomly dropping series when limits are reached. [PR 5788]

When limits are hit, the metrics-generator now produces overflow series labeled metric_overflow="true" instead of discarding data. This ensures no trace-derived metrics are lost. As stale series are pruned, new series automatically emerge from the overflow bucket. [PR 5954]

A new metric, tempo_metrics_generator_registry_active_series_demand_estimate, helps with capacity planning. It uses HyperLogLog estimation to report true cardinality demand even when limits cap the standard active series metric. The estimate has approximately 3% accuracy with minimal memory overhead. [PR 5710]

Additional metrics-generator improvements

This release also includes these metrics-generator updates:

Drop and metric invalid UTF-8 for all metrics in metrics-generator instead of sending invalid data. [PR 5980]
Add database_name_attributes configuration to the service graph processor to support the new db.namespace attribute. [PR 5398]
Enhanced observability of collection failures in the metrics-generator with better error categorization. [PR 5936]
Added validation for metrics-generator histogram buckets to prevent invalid configurations. [PR 5991]
Added configuration to enable instance label for span metrics series, improving observability. [PR 5706]

vParquet5

vParquet5 is now production-ready. vParquet4 remains the default block format in this release.

vParquet5 improves query speed and reduces memory usage with these changes. Dedicated columns now support array values, enabling efficient storage for multi-valued attributes like HTTP headers. [PR 5760, documentation] Virtual span row numbers speed up queries that filter only on resource attributes. [PR 5943]

Dedicated columns now support event-level attributes. A new “blob” detection system identifies high-cardinality attributes like UUIDs and stack traces, then applies optimized encoding to reduce memory pressure and improve performance. The format automatically removes unused dedicated columns to save storage. [PR 5946, documentation]

Flexible dedicated column defaults replace the previous well-known attribute columns. [PR 5696, documentation]

The tempo-cli analyse block(s) command reports on integer attributes to help identify candidates for dedicated columns. [PR 6103, documentation]

vParquet5 also adds native storage for the span:childCount TraceQL intrinsic. [PR 6126, documentation]

Additional vParquet5 updates

This release also includes these vParquet5 improvements and fixes:

Converted ServiceStats data from map to list and updated encoding and compression to enable sorted output and small reduction in file size. [PR 6216]
Fixed issue with orphaned group nodes in vParquet5-preview6. [PR 6095]
Fixed issues related to integer dedicated columns in vParquet5-preview2: [PR 5716]
Fixed disappearing dedicated event attributes in trace view and dedicated blob column validation in vParquet5-preview6. [PR 6100]
Fixed dedicated column array handling to better support attributes with mixed array and non-array values. [PR 6199]

LLM-optimized API responses

Tempo 2.10 introduces an LLM-optimized response format that improves how AI assistants process and understand trace data. When you send the Accept: application/vnd.grafana.llm header to supported endpoints, Tempo returns a simplified JSON structure that flattens the complex OpenTelemetry format into a more readable representation with computed values like duration in milliseconds. The Tempo MCP server (introduced in 2.9) uses this format automatically, so users querying traces through AI assistants like Claude Code or Cursor receive more accurate and useful responses without any configuration changes. This format is currently supported by the trace by ID and tag values endpoints. [PR 5962]

Note
This response format is experimental and subject to change. It’s intended for LLM consumption only and shouldn’t be relied on for programmatic use.

Features and enhancements

The most important features and enhancements in Tempo 2.10 are highlighted below.

TraceQL correctness

Tempo 2.10 adds new TraceQL functions, query capabilities, and related API improvements:

Fixed incorrect results in TraceQL compare() caused by potential hash collision of string array attributes. [PR 5835]
Fixed search by trace:id for short IDs with leading zeros. [PR 5587]
Fixed rare bug where the TraceQL metrics could return an empty response when max series was hit. [PR 6099]
Made top/bottomk deterministic by breaking ties with label values. [PR 5846]
Fixed metrics streaming to release intermediate or incomplete results. [PR 5937]

Performance improvements

Tempo 2.10 includes these performance improvements:

Enhanced TraceQL query performance through various optimizations. [PR 5773]
Improved performance for quantile_over_time() function in TraceQL metrics queries. [PR 5996]
Improved shutdown time in the first 30 seconds for faster graceful shutdowns. [PR 5725]
Added metric for tracking added latency to write requests. [PR 5781]
Improved error message when overrides fail to parse for better debugging. [PR 5787]

User-configurable overrides

The user-configurable overrides API now validates inputs more strictly (for example, rejecting duplicate labels) and exposes more metrics-generator settings for runtime configuration. This helps catch configuration mistakes earlier and fine-tune metrics generation per-tenant without restarts.

Added stricter validation to user-configurable overrides API, such as refusing duplicate labels. [PR 6008, PR 6104]
Exposed metrics-generator configuration options to the user-configurable overrides API, including dimension_mappings, intrinsic_dimensions, trace_id_label_name, and ingestion_time_range_slack. [PR 5989, PR 5974, PR 5972, PR 5958, PR 5973]
Exposed metrics_generator.native_histogram_bucket_factor and metrics_generator.native_histogram_min_reset_duration to the user-configurable overrides API. [PR 5973]
Added ability to disable RetryInfo in per-tenant overrides. [PR 5741]

Other enhancements

This release also includes these enhancements:

Added support for application/protobuf content type in frontend endpoints for improved efficiency. [PR 5865]
Added support for external storage to trace by id endpoint. [PR 6185]
Added validation mode and tests for tempo-vulture. [PR 5605]
Added SSE-C encryption support to S3 backend. [PR 5789]
Added secure connection support to tempo-cli. [PR 5692]
Added default_spans_per_span_set configuration option to control default spans returned per trace in search results. [PR 5858]
Increased weight for heavy TraceQL queries. [PR 5782]
Updated list of intrinsics returned by search tags endpoint. [PR 5857]
Removed MustNewConstMetric to prevent panic and added validation for usage tracker configuration. Added tempo_distributor_usage_tracker_errors_total to surface errors in usage tracker. [PR 5981]
Set maxKeys = 1 for Amazon S3 confirm list operation. [PR 6114]

Upgrade considerations

When upgrading to Tempo 2.10, be aware of these considerations and breaking changes.

Search default result limit changed in 2.10.2

Tempo 2.10.2 sets the default max_result_limit for search to 256*1024. This helps prevent unbounded search responses from returning excessive data by default. [PR 6525]

If your existing workflows rely on larger search result sets, explicitly configure max_result_limit to match your environment and query patterns.

Busybox removed from Tempo image

The Tempo container image no longer includes busybox. This change reduces the image size and attack surface, preventing future busybox-related vulnerabilities from affecting Tempo deployments. [PR 5717]

The image switched from gcr.io/distroless/static-debian12:debug, which includes busybox, to gcr.io/distroless/static-debian12, which doesn’t. The busybox shell and utilities are no longer available inside the running container.

You can no longer exec into the Tempo container with a shell. Commands like kubectl exec -it <pod> -- sh or docker exec -it <container> sh will fail.

To debug a running Tempo container, use one of these alternatives:

Kubernetes ephemeral debug containers (kubectl debug)
Docker Desktop or other container runtime tools that support shell injection for distroless images

If you have custom Docker Compose files or scripts that use the Tempo image for shell operations (such as running chown in an init container), update them to use a separate busybox:latest image for those tasks.

Tempo’s runtime behavior, configuration options, and APIs are unchanged.

vParquet format changes

This release includes breaking changes to vParquet block format support and deprecates older formats ahead of Tempo 3.0.

In preparation for Tempo 3.0, make sure you’re using vParquet4 or higher.

Enable vParquet5 (optional)

vParquet5 is now production-ready and available as an optional upgrade. vParquet4 remains the default block format. [PR 6219]

To enable vParquet5, set the block version in your storage configuration:

storage:
  trace:
    block:
      version: vParquet5

Once enabled, Tempo writes new blocks in vParquet5 format while continuing to read existing vParquet4 blocks. No migration of existing data is required.

vParquet2 removed

vParquet2 encoding has been completely removed from Tempo 2.10. Tempo can no longer read vParquet2 blocks. [PR 6071]

No action is required if you’ve used default settings. The default block format migrated away from vParquet2 several releases ago.

Action is required if your storage configuration explicitly specifies vParquet2. Before upgrading, verify your configuration doesn’t specify vParquet2. If it does:

Update to a previous Tempo release (2.9 or earlier) configured for vParquet3 or higher
Wait for all existing vParquet2 blocks to expire and be deleted from backend storage
Then upgrade to Tempo 2.10

Upgrading to 2.10 while vParquet2 blocks still exist in storage will cause read errors.

vParquet3 and vParquet2 schemas deprecated

TempoDB schemas vParquet3 and vParquet2 are now deprecated and will be removed in Tempo 3.0. [PR 6198]

If you’re using vParquet3, plan your migration to vParquet4 or higher before upgrading to Tempo 3.0.

Breaking changes

Be aware of these breaking changes when upgrading to Tempo 2.10:

Tenant ID validation: Tempo now validates tenant IDs in the frontend and distributor components, using the same validation rules as Grafana Mimir. Previously, Tempo accepted any tenant ID value. Valid tenant IDs may only contain alphanumeric characters (a-z, A-Z, 0-9) and special characters (!, -, _, ., *, ', (, )), with a maximum length of 150 characters. Empty tenant IDs and the values . or .. are rejected. Requests with invalid tenant IDs return an error. [PR 5786]
SearchTagsV2WithRange API change: The Tempo HTTP client SearchTagsV2WithRange function signature has changed. The function previously accepted only start and end parameters. It now requires two additional parameters: scope and query. This change affects Go code that imports the pkg/httpclient package. To maintain existing behavior, pass empty strings for the new parameters: SearchTagsV2WithRange("", "", start, end). [PR 6088]
Go version upgrade: Tempo 2.10 upgrades to Go 1.25.5, which may affect custom builds or deployments with specific Go version requirements. (PRs #5939 #6001, #6096, #6089)

Project Rhythm: New Tempo architecture

Project Rhythm is the codename for the new architecture project for Tempo. The main objective of this project is to address certain trade-offs of the current design that are limiting the ability of Tempo to grow and support new functionalities.

The goals of this project are to:

Eliminate the requirement for replication factor 3 (RF3) -> Support high availability and improve reliability of TraceQL metrics
Decouple the read and write path -> Scalability and reliability
Lay out the foundation to significantly reduce total cost of ownership (TCO) -> Room for growth

Project Rhythm has been discussed in the Tempo Community Calls for several months. Refer to the Tempo Community Calls on YouTube to learn more.

While the new architecture is available in Tempo, it is still considered experimental.

Tempo rearchitecture (experimental)

Refer to the Tempo rearchitecture section in the Tempo 2.10 CHANGELOG for the list of pull requests that are part of this project.

Allow to configure WAL for livestore. [PR 6093]
Added a single binary 3.0 mode --target=all-3.0 to begin testing single binary 3.0 and updating integration tests. This will be removed in 3.0 and become the standard single binary mode. [PR 6021]
Kafka KIP-714 telemetry is enabled by default. Added disable_kafka_telemetry configuration flag to opt out. [PR 6046]
Add query_end_cutoff setting to clamp near-“now” queries (default 30s) for consistent Tempo results. [PR 5682]
Improve dashboards for livestore. [PR 5755]
On startup, the first record for live store to consume is not older than two complete block timeouts. [PR 5693]
Add endpoints for partition downscaling for live-store. [PR 5600, PR 5738, PR 5805, PR 5829]
Add new livestore alert to the tempo-mixin. [PR 5752]
Add partition ownership metric to live-store. [PR 5815]
Simplify block-builder partition assignment with configuration parameter partitions_per_instance. [PR 6022]
Fix block-builder to more precisely validate block encoding on startup. [PR 6037]
Correctly track and reject too large traces in live stores. [PR 5757]
Jsonnet: Correctly add tempo-gossip-member: true labels to block-builders and live-stores. [PR 6125]
Fix wrong sleep duration in block-builder if one of partitions is inactive. [PR 5855]

Security fixes

The following updates were made to address security issues.

2.10.3

S3 SSE-C encryption_key is now treated as a secret to prevent exposure in logs, config dumps, and API responses. The field type changed from a plain string to a redacted secret, matching the existing handling of secret_key and session_token. This affects deployments using S3 SSE-C encryption, which was introduced in Tempo 2.10.0. Addresses CVE-2026-28377. [PR 6711]

2.10.2

Fixed exemplar hint handling in metrics queries to enforce the configured safety cap end-to-end. Addresses CVE-2026-27878. [PR 6559]
Set the default query_frontend.search.max_result_limit to 256*1024 to avoid unbounded search results by default. Addresses CVE-2026-21728. [PR 6525]

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

2.10.2

Applied exemplar hints end-to-end and fixed a safety cap bypass in metrics queries. [PR 6559]
Used frontend MaxExemplars configuration as the single source of truth for exemplar limits and added a safety cap at TraceQL engine entry points. [PR 6515]
Set the default search max_result_limit to 256*1024. [PR 6525]

2.10.1

Corrected avg_over_time calculation. [PR 6252]
Corrected instant query calculation for rate(). [PR 6205]
Fixed query-frontend unable to convert dedicated column blob option. [PR 6377]
Fixed issue with additional matched spans in queries with only resource attributes. [PR 6432]
Fixed live-store deadlock occurring after a complete block failure. [PR 6338]
Fixed query_end_cutoff parameter for query range. [PR 6360]
Fixed dimension_mappings being unconditionally overwritten by overrides, causing them to be set to nil when set through static config but not in overrides. [PR 6390]
Fixed metrics-generator panic when write_relabel_configs is set, caused by uninitialized NameValidationScheme after Prometheus dependency upgrade. [PR 6399]

2.10.0

Prevent slice panic when truncating series after topk() by adding bounds check in metrics query-range combiner. [PR 6010]
Fixed compactor to properly consider SSE-KMS information during metadata copy. [PR 5774]
Fixed spss=0 parameter to properly mean unlimited spans instead of being rejected, and respect max_spans_per_span_set=0 configuration. [PR 5858]
Fixed handle collisions with job and instance labels when targetInfo is enabled. [PR 5780]
Fixed S3 compactor multipart upload to ensure Cloudflare R2 compliance by using uniform chunk sizes. [PR 5838]
Fixed GetTrace() in tempo-query. [PR 5864]
Fixed unsupported nonexistence nil operator for tags lookup. Fixed issue where some tag values/names were not returned due to distinctAttrCollector optimization. [PR 5967]
Fixed delete implementation for S3, GCS, and Azure backends to account for prefix. [PR 6011]
Fixed slow pod startup (~90s) in monolithic mode by returning false from isSharded() when kvstore is empty or in memory. [PR 6090]
Fixed metrics generator to cache failed instances and back off when processor validation or instance creation fails, preventing indefinite retry attempts that could cause out-of-memory errors. [PR 6142]
Fixed panic when trying to compact block with unsupported encodings such as vParquet previews. [PR 6209]
Fixed leading zero handling in native histograms implementations. [PR 6033]

Version 2.9 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.9 release notes

The Tempo team is pleased to announce the release of Tempo 2.9.

This release gives you:

New MCP (Model Context Protocol) server support for LLM integration with tracing data
Enhanced TraceQL metrics with sampling capabilities for improved performance
Significant operational improvements for multi-tenant environments
Numerous TraceQL performance and correctness improvements

These release notes highlight the most important features and bug fixes. For a complete list of changes, refer to the Tempo CHANGELOG.

To learn more about this release, refer to the Tempo 2.9 blog post.

Access tracing data with the Tempo MCP server

Note
Tempo MCP server is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the feature toggle in Grafana to use this feature. Do not enable this feature in production environments.

Tempo 2.9 includes an MCP (Model Context Protocol) server that provides AI assistants and Large Language Models (LLMs) with direct access to distributed tracing data through TraceQL queries and other endpoints. (PR 5212)

MCP is a widely adopted protocol, developed by Anthropic, that standardizes how applications provide context to large language models (LLMs). By integrating MCP with Tempo, you can now leverage LLM-powered tools like Claude Code or Cursor to analyze and derive value from your tracing data. This allows you to better understand interactions between your services and investigate and diagnose issues faster.

Warning
Using this feature will likely cause tracing data to be passed to an LLM or LLM provider. Consider the content of your tracing data and organizational policies when enabling the MCP server.

The feature is disabled by default and can be enabled per tenant for specific use cases.

Refer to MCP server for configuration details and examples.

For more information, refer to LLM-powered insights into your tracing data: introducing MCP support in Grafana Cloud Traces.

TraceQL metrics sampling

TraceQL metrics sampling, using with(sample=true), dynamically and automatically chooses how to sample your tracing data to give you the highest quality signal with examining as little data as possible. (PR 5469)

This sampling method uses an adaptive probabilistic approach that responds to how common spans and traces matching the query are. This sampling is applied at the storage layer, for example, only inspecting xx% spans, or xx% traces, depending on the needs of the query.

When there is a lot of data, it lowers the sampling rate. When matches are rare it keeps the sampling rate higher, possibly never going below 100%. Therefore the performance gain depends on the query.

This behavior can be overridden to focus more on fixed span sampling using with(span_sample=0.xx) or fixed trace sampling using with(trace_sample=0.xx).(PR 5469)

Example:

{} | rate() with(sample=true)
{} | rate() with(span_sample=0.1)

To learn more, refer to TraceQL metrics sampling.

Adaptive sampling was featured in the September 2025 Tempo community call. Watch the recording starting at the 12:00 minute mark to learn more.

Features and enhancements

The most important features and enhancements in Tempo 2.9 are highlighted below.

Operational improvements

These improvements help operators running multi-tenant Tempo deployments improve trace quality and operational visibility.

SLO metrics improvements

Cached querier responses are now excluded from SLO metrics such as inspected bytes, providing more accurate measurements of actual storage and network I/O. This change ensures that performance metrics reflect real resource consumption rather than cached data access. (PR 5185)

Enhanced monitoring and observability

Several new metrics and histograms have been added to improve operational visibility:

Added counter query_frontend_bytes_inspected_total to track total bytes read from disk and object storage. (PR 5310, documentation)
Added histograms spans_distance_in_future_seconds / spans_distance_in_past_seconds that count spans with timestamps outside expected ranges. While spans in the future are accepted, they are invalid and may not be found using the Search API. (PR 4936, documentation)
Added support for scope in cost-attribution usage tracker, allowing more precise tracking with resource. or span. prefixed attributes. (PR 5646, documentation)
Logging and tracing in the write path now includes tenant information. The distributor.ConsumeTraces span status is now properly set to error when trace consumption fails. (PR 5436)

Ingress bytes monitoring

The new metric tempo_distributor_ingress_bytes_total measures bytes received before limits are applied, providing better visibility into incoming data volumes and helping with capacity planning. (PR 5601)

Metrics-generator improvements

The following improvements have been made to the metrics-generator:

The definition of tempo_metrics_generator_processor_service_graphs_expired_edges has been adjusted to exclude edges that are properly counted in the service graph. This provides more accurate metrics for monitoring service graph health and completeness. (PR 5319, documentation)
Invalid Prometheus label names are now automatically dropped in the span metrics processor, preventing metric ingestion errors and ensuring compatibility with Prometheus naming conventions. (PR 5122)
Added support for the new db.namespace attribute for service graph database nodes, aligning with newer OpenTelemetry semantic conventions. This provides better categorization and visualization of database interactions in service graphs. (PR 5602, documentation)
Service graph client virtual nodes now use peer attributes to determine client service names, providing more accurate representation of service-to-service communications in distributed systems. (PR 5381, documentation)

TraceQL correctness and performance improvements

TraceQL metrics have received several important updates that improve correctness and performance. Refer to Upgrade considerations for breaking changes that may affect your deployment.

Improved exemplar selection in quantile_over_time() function provides better representative samples (PR 5278)
Fixed issue preventing very small time steps in metrics queries (PR 5441)
Fixed incorrect TraceQL string comparison for strings starting with numbers (PR 5658)
Fixed incorrect results in TraceQL compare() function for spans with array attributes (PR 5519)
Fixed structural operator behavior with empty left-hand spansets (PR 5578)

Performance improvements

The following improvements have been made to TraceQL query performance:

Performance increase for basic TraceQL metrics queries through reduced overhead in batch processing (PR 5247)
General TraceQL search and metrics performance improvements (PR 5280)
TraceQL performance improvements for complex queries (PR 5218)
Optimized compare() function performance (PR 5419)
TraceQL attribute struct alignment for better memory performance (PR 5240)

Upgrade considerations

When upgrading to Tempo 2.9, be aware of these considerations and breaking changes.

RPM and DEB packages discontinued

Tempo no longer publishes RPM and DEB packages due to an internal change to the handling of signing keys. This can be restored if customers need these packages. (PR 5684)

Migrated Vulture and integration tests to OTLP exporter

In this release, we’ve migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. (PR 5058)

Bucket calculation changes

TraceQL metrics buckets are now calculated based on data in the past instead of the future, which aligns behavior with Prometheus. This resolves issues with empty last buckets that were previously encountered. (PR 5366)

This change may cause differences in existing dashboards and alerts that rely on TraceQL metrics bucket calculations. Review your monitoring and alerting configurations after upgrading.

Series label handling improvements

Fixed incorrect TraceQL metrics results when series labels include both strings and integers with the same textual representation (such as "500" vs 500). The prom_labels field has been removed from TraceQL metrics responses as it was the source of these errors. (PR 5659)

There may be brief interruptions to TraceQL metrics queries during rollout while components run different versions. Plan for a coordinated rollout to minimize impact.

vParquet5 block format

Tempo 2.9 introduces a new experimental vParquet5 block format, designed to improve query performance and reduce storage requirements.vParquet5 has two previews: vParquet5-preview1, low-resolution timestamp columns, and vParquet5-preview2, dedicated integer columns.

Breaking changes are expected before the final release. (PR 5495, PR 5639)

Go version upgrade

Tempo 2.9 upgrades to Go 1.25.0, which may affect custom builds or deployments with specific Go version requirements. (PR 5548)

If you’re building Tempo from source or using custom Docker images, ensure your build environment supports Go 1.25.0.

Project Rhythm: New Tempo architecture

The goals of this project are to:

Eliminate the requirement for replication factor 3 (RF3) -> Support high availability and improve reliability of TraceQL metrics
Decouple the read and write path -> Scalability and reliability
Lay out the foundation to significantly reduce total cost of ownership (TCO) -> Room for growth

Project Rhythm has been discussed in the Tempo Community Calls for several months. Refer to the Tempo Community Calls on YouTube to learn more.

Refer to the Tempo Rearchitecture section in the Tempo 2.9 changelog for the list of pull requests that are part of this project.

Current problems

The current Tempo architecture has the following problems:

Lack of high-availability for reads in TraceQL metric queries: if a generator is down, the data which that generator holds becomes unavailable. This can be improved or mitigated in certain cases, such as rollouts by rolling updates, but never fully solved.
Low write reliability: to reduce the impact that the generators have on the read path, generator ingestion is asynchronous and doesn’t return errors to the client. While this is important to protect the ingester’s write path, it adds unreliability to generator’s ingestion.
High TCO: TraceQL metrics in the generators are based on flushing a new set of blocks of data, in addition to the ingester’s blocks. These two sets of blocks remain separated in compaction, making 2 copies the best case scenario in long-term storage—research shows we’re closer to 3-3.5 in real world scenarios.

Architecture comparison

The current Tempo architecture includes distributors, ingesters, query frontend, queriers, compactors, object storage, and the metrics-generator.

Refer to Tempo architecture for more information.

The diagram illustrates the current Tempo architecture.

Project Rhythm works as follows:

Distributors write incoming requests to Kafka.
Ingesters are substituted by live-stores, a read-path component, replaying from Kafka. Live-stores responsibility is to serve trace queries
A new component, the block-builder, is introduced to the write path. This component is tasked with replaying from Kafka to build blocks which are sent to long-term storage.

Security fixes

The following updates were made to address security issues.

2.9.1

Updated Go to version 1.25.5 to address CVE-2025-61729, CVE-2025-47907, CVE-2025-58183, and CVE-2025-61727. [PR 6089, PR 6096, PR 6233]
Updated golang.org/x/crypto to address CVE-2025-47914, and CVE-2025-58181. [PR 6235]
Updated github.com/expr-lang/expr to v1.17.7 to address CVE-2025-68156. [PR 6234]

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

Fixed Tempo configuration options that are always overridden with configuration overrides section. (PR 5202)
Correctly apply trace idle period in ingesters and add the concept of trace live period. (PR 5346)
Fixed invalid YAML output from /status/runtime_config endpoint by adding document separator. (PR 5371)
Fixed panic in query_range HTTP handling that could be triggered by cancellations or other errors. (PR 5667)
Fixed cache collision for incomplete query in SearchTagValuesV2 (PR 5549)
Deadlock on invalid query to api/v2/search/tags. (SearchTagsV2) (PR 5607)
Fixed incorrect root span detection when spans have a child_of link but no parent. (PR 5557)
Prevent metrics-generator WAL deletion when tenant is empty. (PR 5586)
Fixed docker-compose port configuration for Alloy gRPC (4319 to 4317). (PR 5536)
Fixed panic error from empty span id. (PR 5464)
Return Bad Request from frontend if the provided tag is invalid in SearchTagValuesV2 endpoint. (PR 5493)

Version 2.8 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.8 release notes

The Tempo team is pleased to announce the release of Tempo 2.8.

This release gives you:

New TraceQL features such as most_recent, parent-span filters, sum_over_time, and topk/bottomk to help surface the exact spans and metrics you need.
Concurrent block flushing, iterator optimizations, and back-pressure make ingestion and queries leaner and quicker.
Safer defaults with port 3200, distroless images, Go 1.24, and an upgraded OTLP Collector tighten security and simplify ops.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the Tempo changelog.

Note
Tempo 2.8 has some major upgrade considerations including changing the default http-listen-port from 80 to 3200, removing serverless, and enforcing max attribute size at event, link, and instrumentation scopes.

Refer to Upgrade considerations for details.

Read the Tempo 2.8 blog post for more examples and details about these improvements.

New language features for TraceQL

We’re excited to roll out three powerful enhancements to TraceQL, giving you more flexibility and performance when querying traces in Grafana Cloud with Tempo.

Rank your metrics with new topk(n) and bottomk(n) functions to quickly get your highest and lowest ranking time series.
Aggregate spans over time using sum_over_time() for built-in cumulative sums, such as total bytes, error counts.
Fetch the latest traces first via the experimental most_recent=true query hint.

Ranking `topk` and `bottomk` functions for TraceQL metrics

When you’re looking at latency, error rates, or throughput across hundreds or thousands of services or endpoints, it’s easy to get lost in all the data. Previously, you’d have to pull back the full set of aggregates and then manually inspect or post‑process the results to find your worst offenders or best performers.

With topk(n) and bottomk(n), you can immediately narrow your focus to the top‑ or bottom‑ranked spans in a single, efficient query. This saves time and reduces the data volume you need to scan downstream.

{} | avg_over_time(span:duration) by (span:name) | topk(10)
{} | count_over_time() by (span:name) | bottomk(10)

These are second stage functions used, where:

topk(n) returns the n series with the highest values from a first‑stage aggregation.
bottomk(n) returns the n series with the lowest values.

For more information, refer to the topk and bottomk documentation (PR 4646).

The `sum_over_time` function for TraceQL metrics

The sum_over_time() function lets you aggregate numerical values by computing the sum value of them. The time interval that the sum is computed over is set by the step parameter.

{ span.http.response_content_length > 0 } | sum_over_time(span.http.response_content_length)

With sum_over_time(), you can directly compute cumulative sums inside TraceQL, like total bytes transferred, total error counts, or resource consumption over time.

{}  | sum_over_time(span.http.response_content_length)

For each step interval, sum_over_time(attr) totals the values of attr across all matching spans.

Refer to TraceQL metrics functions for more information. (PR 4786)

Experimental query hint `most_recent=true` to retrieve the most recent traces

When troubleshooting a live incident or monitoring production health, you often need to see the absolute latest traces first. By default, Tempo’s query engine favors speed and returns the first N matching traces, which may not be the newest.

The most_recent hint ensures you’re always looking at the freshest data, so you can diagnose recent errors or performance regressions without missing anything due to early row‑limit cuts.

Examples:

{} with (most_recent=true)
{ span.foo = "bar" } >> { status = error } with (most_recent=true)

With most_recent=true, Tempo performs a deeper search across data shards, retains the newest candidates, and returns traces sorted by start time rather than stopping at the first limit hit. (PR 4238)

The TraceQL query hint most_recent=true can be used with any TraceQL selection query to force Tempo to return the most recent results ordered by time.

For more information, refer to Retrieve most recent results (experimental).

Query by parent span ID

In addition, you can query by parent span id. Using span:parentID, you can ensure that you are looking at a child span of a specific parent, which is useful for example in span linking. Refer to the documentation for more information. (PR 4692)

Major performance and memory usage improvements

Tempo 2.8 removes pooling for large traces. In testing, removing large pooling impacted memory, cutting memory high-water marks to less than half in a high traffic/tenant installation. Some amount of pooling is needed to keep CPU, throughput, and GC rate maintained. Now, we have a fixed 1M default allocation size, which covers most traces.

This graph illustrate the amount of memory used by the compactors. In one cell, we saw major improvements in compactor memory usage after PR 4985 was merged. Refer to PR 4985 for more information, including benchmarks and test results.

Performance improvements

TraceQL and Tempo generally see a number of performance improvements in this release. Refer to benchmarks in the PRs for additional information.

Improve TraceQL performance by reverting EqualRowNumber to an inlineable function. (PR 4705)
Improve iterator performance by using max definition level to ignore parts of the RowNumber while nexting. Nexting is a function multiple in Tempo when you do a big query. Even minor improvements in nexting can have a major impact. (PR 4753)
Increase query-frontend default batch size. TraceQL performance has improved enough to pass larger batches than before and still complete all jobs on one querier. (PR 4844)
Improve Tempo build options. (PR 4755)
Rewrite traces using rebatching. (PR 4690)
Reorder span iterators. (PR 4754)
Improve memcached memory usage by pooling buffers. (PR 4970)

Features and enhancements

This section highlights the most important features and enhancements in Tempo 2.8.

TraceQL correctness

One of our major focuses with this release has been to improve TraceQL’s performance and correctness. The following list summarizes some of the more important improvements.

Fixed behavior for queries like {.foo && true} and {.foo || false}. (PR 4855)
Make comparison to nil symmetric. (PR 4869)
Corrected TraceQL incorrect results for additional spanset filters after a select operation. (PR 4600)
Fixed TraceQL metrics incorrect results for queries with multiple filters that reside in non-dedicated columns that also group by the same variable. (PR 4887)
Fixed metrics streaming for all non-trivial metrics. (PR 4624)
Fixed behavior of {} >> { span.attr-that-doesnt-exist != "foo" }. (PR 5007)
Fixed various edge cases for query range. (PR 4962)
Excluded nestedSetParent and other values from the compare() function. (PR 5196)

The query frontend can cache job results for individual blocks when executing TraceQL. These bug fixes impact caching behavior.

TraceQL results caching bug for floats ending in .0. (PR 4539)
Correctly cache frontend jobs for query range in TraceQL metrics. (PR 4771)
Fixed frontend cache key generation for TraceQL metrics queries to prevent collisions. (PR 5017)

TraceQL exemplar improvements and bugfixes

Tempo 2.8 improves exemplars in TraceQL metrics.

Distribute exemplars over time and hard limit the number of exemplars. (PR 5158)
Right exemplars for histogram and quantiles. (PR 5145)
Fixed for queried number of exemplars. (PR 5115)

Improved error handling and logging

In addition, we’ve updated how Tempo handles errors so it’s easier to troubleshoot.

Updated query range error message. (PR 4929)
Improved rate limit error message when traces size exceeds rate limit. (PR 4986)
Restored dedicated columns logging for completed blocks in the compactor. (PR 4832)
Query-frontend logs add a message to the log line. (PR 4975)

Other enhancements and improvements

This release also has these notable updates.

Added throughput SLO and metrics for the TraceByID endpoint. Configurable via the throughput_bytes_slo field, and it will populate op="traces" label in SLO and throughput metrics. (PR 4668)
Prevented queries in the ingester from blocking flushing traces to disk and memory spikes. (PR 4483)
Host Info Processor now tracks host identifying resource attribute in metric (PR 5152)
Added IPv6 support to the distributor. (PR 4840)
Added default mutex and blocking values. (PR 4979)

Upgrade considerations

When upgrading to Tempo 2.8, be aware of these considerations and breaking changes.

Changed the default listening port

With Tempo 2.8, the default http_listen_port changes from 80 to 3200. Check the configuration options for the server: block in your Tempo configuration file.

server:
    # HTTP server listen host
    [http_listen_address: <string>]

    # HTTP server listen port
    [http_listen_port: <int> | default = 3200]

Refer to issue 4945 for more information for the rationale. PR 4960

Removed Tempo serverless

Tempo serverless has been removed. The following configuration options are no longer valid and should be removed from your Tempo configuration. (PR 4599)

querier:
  search:
    prefer_self: <int>
    external_hedge_requests_at: <duration>
    external_hedge_requests_up_to: <duration>
    external_backend: <string>
    google_cloud_run: <string>
    external_endpoints: <array>

In addition, these Tempo serverless related metrics have been removed: tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total, and tempo_feature_enabled.

Updated, removed, or renamed configuration parameters

Parameter	Comments
`max_span_attr_byte`	Renamed to `max_attribute_bytes`. (PR 4633)
`tempo_discarded_spans_total`	Removed `internal_error` as a reason from `tempo_discarded_spans_total`. (PR 4554)
`tempo_receiver_accepted_span` and `tempo_receiver_refused_spans`	The `name` dimension from `tempo_receiver_accepted_span` and `tempo_receiver_refused_spans` changes from `tempo/jaeger_receiver` to `jaeger/jaeger_receiver`. (PR 4893)

Other upgrade considerations

Upgrade OTEL Collector to v0.122.1. The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver. (PR 4893)
Replace opentracing-contrib/go-grpc by otelgrpc in Tempo query. (PR 4958)
Enforce max attribute size at event, link, and instrumentation scope. The configuration is now per-tenant. Renamed max_span_attr_byte to max_attribute_bytes. (PR 4633)
Converted SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it’s more performant. (PR 4748)
Removed the OpenTelemetry Jaeger exporter, which has been deprecated. (PR 4926)

Security fixes

The following updates were made to address security issues.

2.8.3

Updated Go to version 1.25.5 to address CVE-2025-61729, CVE-2025-47907, CVE-2025-58183, and CVE-2025-61727. [PRs 6089, 6096, 6227]
Updated github.com/expr-lang/expr to v1.17.7 to address CVE-2025-68156. [PRs 6230, 6092]

2.8.2

Updated Go to version 1.24.4. Addresses CVE-2025-22874, CVE-2025-4673, CVE-2025-0913, and GHSA-FV92-FJC5-JJ9H. [PR 5323]

2.8.0

Used distroless base container images for improved security. (PR 4556)
Updated to go 1.24.3. (PR 5110)

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

2.8.3

Fixed deadlock on invalid query to api/v2/search/tags (SearchTagsV2). [PRs 5607, 6228]
Fixed panic in query_range HTTP handling that could be triggered by cancellations or other errors. [PRs 5667, 6229]
Fixed configuration options that were always overridden by runtime overrides, preventing users from setting certain options without using the overrides configuration. [PR 5202]
Fixed trace idle period in ingesters, which was not being correctly applied. Added a max live trace period (default 30s) to prevent very large traces from being held in memory indefinitely when spans are drip-fed over time. [PR 5346]
Fixed the SearchTagValuesV2 endpoint to return 400 Bad Request instead of 500 Internal Server Error when an invalid tag name is provided. [PR 5493]

2.8.2

Added a nil check to partitionAssignmentVar. [PR 5198]
Corrected instant query calculation. [PR 5252]
Fixed tracing context propagation in distributor HTTP write requests. [PR 5312]
Fixed search by trace:id with short trace ID. [PR 5331]
Fixed bug where most_recent=true wouldn’t return most recent results when query overlapped ingesters and few other blocks. [PR 5438]
Fixed a panic when the counter series was missing during avg_over_time aggregation. [PR 5300]

2.8.1

Fixed an ingester issue where a hash collision could lead to spans stored incorrectly. (PR 5276)

2.8.0

Chose a default step for a gRPC streaming query range request if none is provided. Correctly copy exemplars for metrics like | rate() when gRPC streaming. (PR 4576)
Fixed distributor issue where a hash collision could lead to spans stored incorrectly (PR 5186)
Added object name to cache key in ReadRange (PR 4982)
Fixed rare panic during compaction (PR 4915)
Returned the operand as the only value if the tag is already filtered in the query (PR 4673)
Included cost attribution when converting from default configuration to legacy one. (PR 4937)
Updated memcached to respect cancelled context to prevent panic. (PR 5041)
Fixed setting processors in user configurations overrides via API. (PR 4741)
Fixed panic on startup. (PR 4744)
Fixed intrinsic tag lookups dropped when max tag lookup response size is exceeded. (PR 4784)
Fixed error propagation in the SyncIterator. (PR 5045)
Fixed mixin to include otlp_v1_traces HTTP write route (PR 5072)
Fixed TempoBlockListRisingQuickly alert grouping. (PR 4876)
Fixed metrics generator host info processor overrides configuration. (PR 5118)
Fixed metrics generator target_info to skip attributes with no name to prevent downstream errors. (PR 5148)

Version 2.7 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.7 release notes

The Tempo team is pleased to announce the release of Tempo 2.7.

This release gives you:

Ability to precisely track ingested traffic and attribute costs based on custom labels
A series of enhancements that significantly boost Tempo performance and reduce its overall resource footprint.
New TraceQL capabilities
Improvements to TraceQL metrics

Read the Tempo 2.7 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the Tempo changelog.

Features and enhancements

The most important features and enhancements in Tempo 2.7 are highlighted below.

Note
This release changes gRPC compression to snappy by default to improve performance. Refer to gRPC compression for more information.

Track ingested traffic and attribute costs

This new feature lets tenants precisely measure and attribute costs for their ingested trace data by leveraging custom labels. This functionality provides a more accurate alternative to existing size-based metrics and meets the growing need for detailed cost attribution and billing transparency.

Modern organizations are increasingly reliant on distributed traces for observability, yet reconciling the costs associated with different teams, services, or departments can be challenging. The existing size metric isn’t accurate enough by missing non-span data and can lead to under- or over-counting. The new usage tracking feature overcomes these issues by splitting resource-level data fairly and providing up to 99% accuracy—perfect for cost reconciliation.

Unlike the previous method, this new feature precisely accounts for every byte of trace data in the distributor—the only Tempo component with the original payload. A new API endpoint, /usage_metrics, exposes the per-tenant metrics on ingested data and cost attribution, and can be controlled with per-tenant configuration.

This feature is designed as a foundation for broader usage tracking capabilities, where additional trackers will allow organizations to measure and report on a range of usage metrics. (#4162)

For additional information, refer to the Usage metrics documentation.

Major performance and memory usage improvements

We’re excited to announce a series of enhancements that significantly boost Tempo performance and reduce its overall resource footprint.

Better refuse large traces: The ingester now reuses generator code to better detect and reject oversized traces. This change makes trace ingestion more reliable and prevents capacity overloads. Plus, two new metrics tempo_metrics_generator_live_trace_bytes and tempo_ingester_live_trace_bytes provide deeper visibility into per-tenant byte usage. (#4365)

Reduced allocations: We’ve refined how the query-frontend handles incoming traffic to eliminate unnecessary allocations when query demand is low. As part of these improvements, the querier_forget_delay configuration option has been removed because it no longer served a practical purpose. (#3996) This release also reduces the ingester working set by improving prelloc behavior. It also adds tunable prealloc env variables PREALLOC_BKT_SIZE, PREALLOC_NUM_BUCKETS, PREALLOC_MIN_BUCKET, and metric tempo_ingester_prealloc_miss_bytes_total to observe and tune prealloc behavior. (#4344, #4369)

Faster tag lookups and collector operations: Multiple optimizations (#4100, #4104, #4109) make tag lookups and collector tasks more responsive, particularly for distinct value searches. Additionally, enabling disk caching for completed blocks in ingesters significantly cuts down on query latency and lowers overall I/O overhead. (#4069)

Reduce goroutines in non-querier components: A simplified design across non-querier components lowers the total number of goroutines, making Tempo more efficient and easier to scale—even with high trace volumes. (#4484)

New TraceQL capabilities

New in Tempo 2.7, TraceQL now allows you to query the instrumentation scope fields (#3967), letting you filter and explore your traces based on where and how they were instrumented.

We’ve extended TraceQL to automatically collect matches from array values (#3867), making it easier to parse spans containing arrays of attributes.

Query times are notably faster, thanks to multiple optimizations (#4114, #4163, #4438). Whether you’re running standard queries or advanced filters, you should see a significant speed boost.

Tempo now uses the Prometheus “fast regex” engine to accelerate regular expression-based filtering (#4329). As part of this update, all regular expressions matches are now fully anchored. This breaking change means span.foo =~ "bar" is evaluated as span.foo =~ "^bar$". Update any affected queries accordingly.

Query improvements

In API v2 queries, Tempo can now return partial traces even when they exceed the max bytes limit (#3941). This ensures you can still retrieve and inspect useful segments of large traces to aid in debugging. Refer to the query v2 API endpoint documentation for more information.

TraceQL metrics improvements (experimental)

In TraceQL metrics, we’ve added a new avg_over_time function (#4073) to help you compute average values for trace-based metrics over specified time ranges, making it simpler to spot trends and anomalies.

Tempo now supports min_over_time (#3975) and max_over_time (#4065) queries, giving you more flexibility in analyzing the smallest or largest values across your trace data.

For more information about TraceQL metrics, refer to TraceQL metrics functions.

Other enhancements and improvements

This release also has these notable updates.

2.7.1

Default to snappy compression for all gRPC communications internal to Tempo. We feel this is a nice balance of resource usage and network traffic. For a discussion on alternatives, refer to this discussion thread. (#4696)

2.7.0

The metrics-generator introduces a generous limit of 100 for failed flush attempts (#4254) to prevent constant retries on corrupted blocks. A new metric also tracks these flush failures, offering better visibility into potential issues during ingestion.
For span metrics, the span multiplier now also sources its value from the resource attributes (#4210). This makes it possible to adjust and correct metrics using service or environment configuration, ensuring more accurate data reporting.
The tempo-cli now supports dropping multiple trace IDs in a single command, speeding up administrative tasks and simplifying cleanup operations. (#4266)
An optional log, log_discarded_spans, tracks spans discarded by Tempo. This improves visibility into data ingestion workflows and helps you quickly diagnose any dropped spans. (#3957)
You can now impose limits on tag and tag-value lookups, preventing runaway queries in large-scale deployments and ensuring a smoother user experience. (#4320)
The tags and tag-values endpoints have gained new throughput and SLO-related metrics. Get deeper insights into query performance and reliability straight from Tempo’s built-in monitoring. (#4148)
Tempo now exposes a SemVer version in its /api/status/buildinfo endpoint, providing clear visibility into versioning, particularly useful for cloud deployments and automated pipelines. (#4110)

Upgrade considerations

When upgrading to Tempo 2.7, be aware of these considerations and breaking changes.

OpenTelemetry Collector receiver listens on `localhost` by default

After this change, the OpenTelemetry Collector receiver defaults to binding on localhost rather than 0.0.0.0. Tempo installations running in Docker or other container environments must update their listener address to continue receiving data. (#4465)

Most Tempo installations use the receivers with the default configuration:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

This used to work fine since the receivers defaulted to 0.0.0.0:4317 and 0.0.0.0:4318 respectively. With the changes to replace unspecified addresses, the receivers now default to localhost:4317 and localhost:4318.

As a result, connections to Tempo running in a Docker container won’t work anymore.

To workaround this, you need to specify the address you want to bind to explicitly. For instance, if Tempo is running in a container with hostname tempo, this should work:

# ...
http:
  endpoint: 'tempo:4318'

You can also explicitly bind to 0.0.0.0 still, but this has potential security risks:

# ...
http:
  endpoint: '0.0.0.0:4318'

Tempo serverless deprecation

Tempo serverless is now officially deprecated and will be removed in an upcoming release. Prepare to migrate any serverless workflows to alternative deployments. (#4017, documentation)

There are no changes to this release for serverless. However, you’ll need to remove these configurations before the next release.

Anchored regular expression matchers in TraceQL

TraceQL now uses the Prometheus “fast regex” engine to accelerate regular expression-based filtering (#4329). As part of this update, all regular expression matches are now fully anchored. This breaking change means span.foo =~ "bar" is evaluated as span.foo =~ "^bar$". Update any affected queries accordingly.

For more information, refer to the Comparison operators TraceQL documentation.

Migration from OpenTracing to OpenTelemetry

This release removes the use_otel_tracer option. Configure your spans via standard OpenTelemetry environment variables. For Jaeger exporting, set OTEL_TRACES_EXPORTER=jaeger.For more information, refer to the OpenTelemetry documentation. (#3646)

Added, updated, removed, or renamed configuration parameters

Parameter	Comments
`querier_forget_delay`	Removed. The `querier_forget_delay` setting provided no effective functionality and has been dropped. (#3996)
`use_otel_tracer`	Removed. Configure your spans via standard OpenTelemetry environment variables. For Jaeger exporting, set `OTEL_TRACES_EXPORTER=jaeger`. (#3646)
`max_spans_per_span_set`	Added to query-frontend configuration. The limit is enabled by default and set to 100. Set it to `0` to restore the old behavior (unlimited). Otherwise, spans beyond the configured max are dropped. (#4275)
`use_otel_tracer`	The `use_otel_tracer` option is removed. Configure your spans via standard OpenTelemetry environment variables. For Jaeger exporting, set `OTEL_TRACES_EXPORTER=jaeger`. (#3646)

gRPC compression set to snappy

Tempo 2.7.1 sets gRPC compression between all components to be snappy. Using snappy provides a balanced approach to compression between components that will work for most installations.

If you prefer a different balance of CPU/Memory and bandwidth, consider disabling compression or using zstd.

For a discussion on alternatives, refer to this discussion thread. (#4696).

gRPC compression disabled

Tempo 2.7.0 release disabled gRPC compression in the querier and distributor for performance reasons. (#4429). Our benchmark suggests that without compression, queriers and distributors use less CPU and memory.

Tempo 2.7.1 changed the default value to snappy for internal components.

Note
While disabling gRPC compression improves performance, it will increase data usage and network traffic, which can impact cloud billing depending on your configuration.

If you notice increased network traffic or issues, check the gRPC compression settings. You can enable gRPC compression using snappy:

ingester_client:
  grpc_client_config:
    grpc_compression: 'snappy'
metrics_generator_client:
  grpc_client_config:
    grpc_compression: 'snappy'
querier:
  frontend_worker:
    grpc_client_config:
      grpc_compression: 'snappy'

Refer to gRPC compression configuration for more information.

Other upgrade considerations

The Tempo CLI now targets the /api/v2/traces endpoint by default. Use the --v1 flag if you still rely on the older /api/traces endpoint. (#4127)
If you already set the X-Scope-OrgID header in per-tenant overrides or global Tempo configuration, it’s now honored and not overwritten by Tempo. This may change behavior if you previously depended on automatic injection. (#4021)
The AWS Lambda build output changes from main to bootstrap. Follow the AWS migration steps to ensure your Lambda functions continue to work. (#3852)

Bugfixes

For a complete list, refer to the Tempo CHANGELOG.

2.7.2

Fix rare panic that occurred when a querier modified results from ingesters and generators while they were being marshalled to proto. This bug also impacted query correctness on recent trace data by returning incomplete results before they were ready. (#4790)

2.7.0

Add invalid_utf8 to reasons span metrics will discard spans. (#4293) We now catch values in your tracing data that can’t be used as valid metrics labels. If you want span metrics by foo, but foo has illegal prom changes in it, then they won’t be written.
Metrics-generators: Correctly drop from the ring before stopping ingestion to reduce drops during a rollout. (#4101)
Correctly handle 400 Bad Request and 404 Not Found in gRPC streaming. (#4144)
Correctly handle Authorization header in gRPC streaming. (#4419)
Fix TraceQL metrics time range handling at the cutoff between recent and backend data. (#4257)
Fix several issues with exemplar values for TraceQL metrics. (#4366, #4404)
Utilize S3Pass and S3User parameters in tempo-cli options, which were previously unused in the code. (#4259)

Version 2.6 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.6 release notes

The Tempo team is pleased to announce the release of Tempo 2.6.

This release gives you:

Additions to the TraceQL language, including the ability to search by span events, links, and arrays
Additions to TraceQL metric query-types including a compare function and the ability to do instant queries (which will return faster than range queries).
Performance and stability enhancements

Read the Tempo 2.6 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the Tempo changelog.

Features and enhancements

The most important features and enhancements in Tempo 2.6 are highlighted below.

Additional TraceQL metrics (experimental)

In this release, we’ve added several TraceQL metrics. In Tempo 2.6, TraceQL metrics adds:

Exemplars [PR 3824, documentation]
Instant metrics queries using /api/metrics/query [PR 3859, documentation]
A q parameter to tag-name filtering the search v2 API [PR 3822, documentation]
A new compare() metrics function [PR 3695, documentation]

Additionally, we’re working on refactoring the replication factor. Refer to the Operational change for TraceQL metrics section for details.

Note that using TraceQL metrics may require additional system resources.

For more information, refer to the TraceQL metrics queries and Configure TraceQL metrics.

TraceQL improvements

Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, refer to the TraceQL documentation.

We’ve added event attributes and link scopes. Like spans, they both have instrinsics and attributes.

The event scope lets you query events that happen within a span. A span event is a unique point in time during the span’s duration. While spans help build the structural hierarchy of your services, span events can provide a deeper level of granularity to help debug your application faster and maintain optimal performance. To learn more about how you can use span events, read the What are span events? blog post. [PRs 3708, 3708, 3908]

If you’ve instrumented your traces for span links, you can use the link scope to search for an attribute within a span link. A span link associates one span with one or more other spans. [PRs 3814, 3741]

For more information on span links, refer to the Span Links documentation in the Open Telemetry project.

You can search for an attribute in your link:

{ link.opentracing.ref_type = "child_of" }

We’ve also added autocomplete support for events and links. [PR 3846]

Tempo 2.6 improves TraceQL performance with these updates:

Performance improvement for rate() by () queries [PR 3719]
Add caching to query range queries [PR 3796]
Only stream diffs on metrics queries [PR 3808]
Tag value lookup use protobuf internally for improved latency [PR 3731]
TraceQL metrics queries use protobuf internally for improved latency [PR 3745]
TraceQL search and other endpoints use protobuf internally for improved latency and resource usage [PR 3944]
Add local disk caching of metrics queries in local-blocks processor [PR 3799]
Performance improvement for queries using trace-level intrinsics [PR 3920]
Use multiple goroutines to unmarshal responses in parallel in the query frontend. [PR 3713]

Native histogram support

The metrics-generator can produce native histograms for high-resolution data. PR 3789

Native histograms are a data type in Prometheus that can produce, store, and query high-resolution histograms of observations. It usually offers higher resolution and more straightforward instrumentation than classic histograms.

To learn more, refer to the Native histogram documentation.

Performance improvements

One of our major improvements in Tempo 2.6 is the reduction of memory usage due to polling improvements. [PRs 3950, 3951, 3952]

This improvement is a result of some of these changes:

Add data quality metric to measure traces without a root. [PR 3812]
Reduce memory consumption of query-frontend. [PR 3888]
Reduce allocations of caching middleware. [PR 3976]
Reduce allocations building queriers sharded requests. [PR 3932]
Improve trace id lookup from Tempo Vulture by selecting a date range. [PR 3874]

Other enhancements and improvements

This release also has these notable updates.

2.6.1

Register gRPC health server to tempo-query. [PR 4178]
Support Tempo on IBM s390x. [PR 4175]
tempo-query: Separate TLS settings for server and client. [PR 4177]
Speedup tempo-query trace search by allowing parallel queries. [PR 4159]

2.6.0

Bring back OTel receiver metrics. [PR 3917]
Add a q parameter to /api/v2/search/tags for tag name filtering. [PR 3822]
Add middleware to block matching URLs. [PR 3963]
Add data quality metric to measure traces without a root. [PR 3812]
Implement polling tenants concurrently. [PR 3647]
Add native histograms for internal metrics [PR 3870]
Add a Tempo CLI command to drop traces by id by rewriting blocks. [PR 3856, documentation]
Add new OTel compatible Traces API V2. [PR 3912, documentation]
Rename Batches to ResourceSpans. [PR 3895]

Upgrade considerations

When upgrading to Tempo 2.6, be aware of these considerations and breaking changes.

Operational change for TraceQL metrics

We’ve changed to an RF1 (Replication Factor 1) pattern for TraceQL metrics as we were unable to hit performance goals for RF3 de-duplication. This requires some operational changes to query TraceQL metrics.

TraceQL metrics are still considered experimental. We hope to mark them GA soon when we productionize a complete RF1 write-read path. [PRs 3628, 3691, 3723, 3995]

For recent data

The local-blocks processor must be enabled to start using metrics queries like { } | rate(). If not enabled metrics queries fail with the error localblocks processor not found. Enabling the local-blocks processor can be done either per tenant or in all tenants.

Per-tenant in the per-tenant overrides:

overrides:
  'tenantID':
    metrics_generator_processors:
      - local-blocks

By default, for all tenants in the main config:

overrides:
  defaults:
    metrics_generator:
      processors: [local-blocks]

Add this configuration to run TraceQL metrics queries against all spans (and not just server spans):

metrics_generator:
  processor:
    local_blocks:
      filter_server_spans: false

For historical data

To run metrics queries on historical data, you must configure the local-blocks processor to flush RF1 blocks to object storage:

metrics_generator:
  processor:
    local_blocks:
      flush_to_storage: true

Transition to vParquet4

vParquet4 format is now the default block format. It’s production ready and we highly recommend switching to it for improved query performance. [PR 3810]

Upgrading to Tempo 2.6 modifies the Parquet block format. You don’t need to do anything with Parquet to go from 2.5 to 2.6. If you used vParquet2 or vParquet3, all of your old blocks remain and can be read by Tempo 2.6. Tempo 2.6 creates vParquet4 blocks by default, which enables the new TraceQL features.

Although you can use Tempo 2.6 with vParquet2 or vParquet3, you can only use vParquet4 with Tempo 2.5 and later. If you are using 2.5 with vParquet4, you’ll need to upgrade to Tempo 2.6 to use the new TraceQL features.

You can also use the tempo-cli analyse blocks command to query vParquet4 blocks. PR 3868]. Refer to the Tempo CLI documentation for more information.

For information on upgrading, refer to Upgrade to Tempo 2.6 and Choose a different block format.

Updated, removed, or renamed configuration parameters

Parameter	Comments
`storage: azure: use_v2_sdk:`	Removed. Azure v2 is the only and primary Azure backend [PR 3875]
`autocomplete_filtering_enabled`	The feature flag option has been removed. The feature is always enabled. [PR 3729]
`completedfilepath` and `blocksfilepath`	Removed unused WAL configuration options. [PR 3911]
`compaction_disabled`	New. Allow compaction disablement per-tenant. [PR 3965, documentation]
`Storage: s3: [enable_dual_stack: <bool>]`	Boolean flag to activate or deactivate dualstack mode on the Storage block configuration for S3. [PR 3721, /configuration/#standard-overrides">documentation]

Other breaking changes

BREAKING CHANGE tempo-query is no longer a Jaeger instance with grpcPlugin. It’s now a standalone server. Serving a gRPC API for Jaeger on 0.0.0.0:7777 by default. [PR 3840]

Bugfixes

For a complete list, refer to the Tempo CHANGELOG.

2.6.1

Bring back application-json content-type header. [PR 4123]

2.6.0

Fix panic in certain metrics queries using rate() with by. [PR 3847]
Fix metrics queries when grouping by attributes that may not exist. [PR 3734]
Fix metrics query histograms and quantiles on traceDuration. [PR 3879]
Fix divide by 0 bug in query frontend exemplar calculations. [PR 3936]
Fix autocomplete of a query using scoped instrinsics. [PR 3865]
Improved handling of complete blocks in the localblocks processor after enabling flushing. [PR 3805]
Fix double appending the primary iterator on second pass with event iterator. [PR 3903]
Fix frontend parsing error on cached responses [PR 3759]
max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit. [PR 3618]
Fix HTTP connection reuse on GCP and AWS by reading io.EOF through the http body. [PR 3760]
Handle out of boundaries spans kinds. [PR 3861]
Maintain previous tenant blocklist on tenant errors. PR 3860
Fix prefix handling in Azure backend Find() call. [PR 3875]
Correct block end time when the ingested traces are outside the ingestion slack. [PR 3954]
Fix race condition where a streaming response could be marshaled while being modified in the combiner resulting in a panic. [PR 3961]
Pass search options to the backend for SearchTagValuesBlocksV2 requests. [PR 3971]

Version 2.5 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.5 release notes

The Tempo team is pleased to announce the release of Tempo 2.5.

This release gives you:

New features, including more TraceQL metrics, gRPC streaming endpoints, and new experimental vParquet4 block format
TraceQL enhancements and performance improvements
Performance and stability enhancements

As part of this release, we’ve updated ownership for /var/tempo from root:root to a new tempo:tempo user with a UUID of 10001. Learn about this breaking change in Upgrade considerations.

Read the Tempo 2.5 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bug fixes. For a complete list, refer to the Tempo CHANGELOG.

Features and enhancements

The most important features and enhancements in Tempo 2.5 are highlighted below.

Additional TraceQL metrics (experimental)

In this release, we’ve added several TraceQL metrics. Tempo 2.4 introduced the rate() function to view rates of spans. For this release, we’ve added quantile_over_time and histogram_over_time. PR 3605, PR 3633, PR 3644]

You can use quantiles_over_time allowing users to aggregate numerical values, such as the all-important span duration. Notice that you can specify multiple quantiles in the same query.

You can use historgram_over_time to display the metrics as a heatmap.

Note that using TraceQL metrics may require additional system resources.

For more information, refer to the TraceQL metrics queries and Configure TraceQL metrics.

gRPC streaming endpoints for Tempo APIs

With this feature, you can now see partial query results as they come in, so you no longer have to wait for the whole query to finish. This is perfect for big queries that take a long time to return a response.

The Tempo API endpoints now support gRPC streaming for tag queries and metrics. We’ve added new streaming endpoints for:

SearchTags
SearchTagsV2
SearchTagValues
SearchTagValuesV2
MetricsQueryRange

These new streaming endpoints are not supported by Grafana yet. However, they are supported by the Tempo CLI tool.

In the Tempo CLI, you can use the --use-grpc option to enable GRPC streaming.

To learn more, refer to the Tempo gRPC API and Tempo CLI documentation. PR 3460 [PR #3584]

Note
Streaming over HTTP requires the stream_over_http_enabled flag to be set. For more information, refer to Tempo GRPC API documentation.

In addition, we’ve reduced memory consumption in the frontend for large traces. [PR 3522]

New vParquet4 block format (experimental)

New in Tempo 2.5, the vParquet4 block format is required for querying links, events, and arrays and improves query performance relative to previous formats. [PR 3368]

In addition, we’ve updated the OTLP schema to add attributes to instrumentation scope in vParquet4.[PR 3649]

While you can use vParquet4, keep in mind that it’s experimental. If you choose to use vParquet4 and then opt to revert to vParquet3, any vParquet4 blocks would not be readable by vParquet3.

To try vParquet4, refer to Choose a block format.

TraceQL

Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, see the TraceQL documentation.

For information on planned future extensions to the TraceQL language, refer to future work.

We’ve made the following improvements to TraceQL:

Add support for scoped intrinsics using a colon (:). The available scoped intrinsics are trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage. [[PR 3629]
Performance improvements on TraceQL and tag value search. [PR 3650,PR 3667]
Add nestedSetLeft/Right/Parent intrinsics to TraceQL. [PR 3497]
Reduce memory usage and improve performance when working with large datasets with many repeated strings. [PR 3411]
Add support querying by trace:id and span:id [PR 3670]

Other enhancements and improvements

Add TLS support for Memcached Client [PR 3585]
Remove hardcoded delay in distributor shutdown [PR 3687]
Surface new labels for uninstrumented services and systems [PR 3543]
Add messaging-system latency histogram to service-graph [PR 3453]
Add support for sharded ingester queries [PR 3574]
Better compaction throughput and memory usage [PR 3579]
Return a less confusing error message to the client when refusing spans due to ingestion rates. [PR 3485]
Clean Metrics Generator’s Prometheus WAL before creating instance [PR 3548]
Delete any remaining objects for empty tenants after a configurable duration, requires configuration enable PR 3611]

Upgrade considerations

When upgrading to Tempo 2.5, be aware of these considerations and breaking changes.

Docker image runs as new UID 10001

The Tempo process in the official Docker image used to run as root. The Tempo process now runs as UID 10001 in the Docker image.

Components such as ingesters and metrics generators that maintain files on disk won’t come up cleanly without intervention. The new user 10001 won’t have access to the old files created by root.

The ownership of /var/tempo changed from root:root to tempo:tempo with the UID/GID of 10001.

The ingester and metrics-generator statefulsets may need to run chown to change ownership to start properly.

Refer to PR 2265 to see a Jsonnet example of an init container.

This change doesn’t impact you if you used the Helm chart with the default security context set in the chart. All data should be owned by the tempo user already. The UID won’t impact Helm chart users.

Support for vParquet format removed

The original vParquet format has been removed from Tempo 2.5. Direct upgrades from Tempo 2.1 to Tempo 2.5 aren’t possible. You will need to upgrade to an intermediate version and wait for the old vParquet blocks to fall out of retention before upgrading to 2.5. PR 3663]

vParquet(1) won’t be recognized as a valid encoding and any remaining vParquet(1) blocks won’t be readable. Installations running with historical defaults shouldn’t require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet(1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.

Updated, removed, or renamed configuration parameters

Parameter	Comments
`[hedge_requests_at: <duration> \| default = 2s ]` `[hedge_requests_up_to: <int> \| default = 2 ]`	Removed options from the configuration. [PR #3522]

Additional considerations

Updating to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers. PR 3649]
Removes the addition of a tenant in multi-tenant trace id lookup. PR 3522]

Bugfixes

For a complete list, refer to the Tempo changelog.

Fix handling of regex matchers in autocomplete endpoints PR 3641
Return unfiltered results when a bad TraceQL query is provided in autocomplete. PR 3426
Add support for dashes, quotes and spaces in attribute names in autocomplete PR 3458
Fix metrics query results when filtering and rating on the same attribute PR 3428
Fix metrics query results when series contain empty strings or nil values PR 3429
Fix metrics query duration check, add per-tenant override for max metrics query duration PR 3479
Fix metrics query panic “index out of range [-1]” when a trace has zero-length ID PR 3668
Correctly handle 429s in GRPC search streaming. PR 3469
Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. PR 3443
Add spss and limit to the frontend cache key to prevent the return of incorrect results. PR 3557
Use OS path separator to split blocks path. PR 3552
Correctly parse TraceQL queries with > 1024 character attribute names or static values. PR 3571
Fix span-metrics’ subprocessors bug that applied wrong configurations when running multiple tenants. PR 3612
Fix panic in query-frontend when combining results PR 3683
Fix TraceQL queries involving non boolean operations between statics and attributes. PR 3698

Version 2.4 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.4 release notes

The Tempo team is pleased to announce the release of Tempo 2.4.

This release gives you:

New features, including multi-tenant queries and experimental TraceQL metrics queries
Performance enhancements, thanks to the addition of new caching tiers
Cost savings, thanks to polling improvements that reduce calls to object storage

As part of this release, vParquet3 has also been promoted to the new default storage format for traces. For more about why we’re so excited about vParquet3, refer to Accelerate TraceQL queries at scale with dedicated attribute columns in Grafana Tempo.

Read the Tempo 2.4 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bug fixes. For a complete list, refer to the Tempo CHANGELOG.

Features and enhancements

The most important features and enhancements in Tempo 2.4 are highlighted below.

Multi-tenant queries

Tempo now allows you to query multiple tenants at once. We’ve made multi-tenant queries compatible with streaming (first released in v2.2) so you can get query results as fast as possible. To learn more, refer to Cross-tenant federation and Enable multi-tenancy. [PRs 3262, 3087]

TraceQL metrics (experimental)

We’re excited to announce the addition of metrics queries to the TraceQL language. Metric queries extend trace queries by applying a function to trace query results. This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.

In this case, we’re calculating the rate of the erroring spans coming from the service foo. Rate is a spans/sec quantity.

{ resource.service.name = "foo" && status = error } | rate()

In addition, you can use Grafana Explore to query and visualize the metrics with the Tempo data source in Grafana or Grafana Cloud.

For more information, refer to the TraceQL metrics documentation. [PRs 3227 #3252, 3258]

To learn more about the TraceQL syntax, see the TraceQL documentation.

TraceQL performance improvements

We continue to make query performance improvements so you spend less time waiting on results to your TraceQL queries. Below are some notable PRs that made it into this release:

Improve TraceQL regular expression performance in certain queries. [PR 3139]
Improve TraceQL performance in complex queries. [PR 3113]
TraceQL/Structural operators performance improvement. [PR 3088]

vParquet3 is now the default block format

Tempo 2.4 makes vParquet3 the default storage format.

We’re excited about vParquet3 relative to prior formats because of its support for dedicated attribute columns, which help speed up queries on your largest and most queried attributes. We’ve seen excellent performance improvements when running it ourselves, and by promoting it to the default, we’re signaling that it’s ready for broad adoption.

Dedicated attribute columns, available using vParquet3, improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list. For more information, refer to Accelerate TraceQL queries at scale with dedicated attribute columns in Grafana Tempo. [PR 2526]

If you had manually configured vParquet3, we recommend removing it to move forward with Tempo defaults.

To read more about the design of vParquet3, refer to the design proposal. For general information, refer to the Apache Parquet schema.

Additional caching layers

Tempo has added two new caches to improve TraceQL query performance. The frontend-search cache handles job search caching. The parquet-page cache handles page level caching. Refer to the Cache section of the Configuration documentation for how to configure these new caching layers.

As part of adding these new caching layers, we’ve refactored our caching interface. This includes breaking changes described in Breaking Changes. [PRs 3166, 3225, 3196]

Improved polling for cost reduction

We’ve improved how Tempo polls object storage, ensuring that we reuse previous results. This has dramatically reduced the number of requests Tempo makes to the object store. Not only does this reduce the load on your object store, for many, it will save you money (since most hosted object storage solutions charge per request).

We’ve also added the list_blocks_concurrency parameter to allow you to tune the number of list calls Tempo makes in parallel to object storage so you can select the value that works best for your environment. We’ve set the default value to 3, which should work well for the average Tempo cluster. [PR 2652]

Other enhancements and improvements

In addition, the following improvements have been made in Tempo 2.4:

Improved Tempo error handling on writes, so that one erroring trace doesn’t result in an entire batch of traces being dropped. PR 2571
Added per-tenant compaction window. PR 3129
Added --max-start-time and --min-start-time flag to tempo-cli command analyse blocks. PR 3250
Added per-tenant configurable remote_write headers to metrics-generator. #3175
Added variable expansion support to overrides configuration. PR 3175
Added HTML pages /status/overrides and /status/overrides/{tenant}. PR 3244 #3332
Precalculate and reuse the vParquet3 schema before opening blocks. PR 3367
Made the trace ID label name configurable for remote written exemplars. PR 3074
Performance improvements in span filtering. PR 3025
Introduced localblocks process configuration option to select only server spans. PR 3303

Upgrade considerations

When upgrading to Tempo 2.4, be aware of these considerations and breaking changes.

Transition to vParquet 3

vParquet3 format is now the default block format. It’s production ready and we highly recommend switching to it for improved query performance and dedicated attribute columns.

Upgrading to Tempo 2.4 modifies the Parquet block format. Although you can use Tempo 2.3 with vParquet2 or vParquet3, you can only use Tempo 2.4 with vParquet3.

With this release, the first version of our Parquet backend, vParquet, is being deprecated. Tempo 2.4 still reads vParquet1 blocks. However, Tempo will exit with error if they’re manually configured. [PR 3377]

For information on upgrading, refer to Upgrade to Tempo 2.4 and Choose a different block format .

Updated, removed, or renamed configuration parameters

Parameter	Comments
`autocomplete_filtering_enabled`	Set to `true` by default [PR 3178]
`distributor.log_received_traces`	Use the `distributor.log_received_spans` configuration block instead. [PR #3008]
`tempo_query_frontend_queries_total{op="searchtags\|metrics"}`	Removed deprecated frontend metrics configuration option
YAML `storage: trace: cache: search: cache_control: background_cache: memcached:`	These fields have been removed in favor of the new cache configuration. Refer to Cache configuration refactored.

The distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors. The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics.

Removed experimental websockets support for search streaming

GPRC is now the supported method for streaming results. Websockets support for search streaming has been removed. Websocket support was initially added due to conflicts with GRPC, HTTP, and TLS. Those issues were corrected in PR 3300. PR 3307

Cache configuration refactored

The major cache refactor allows multiple role-based caches to be configured. [PR 3166] This change resulted in several fields being deprecated (refer to the old configuration). These have all been migrated to a top level cache: field.

For more information about the configuration, refer to the Cache section.

The old configuration block looked like this:

storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:

With the new configuration, you create your list of caches with either redis or memcached cluster with your configuration, then define the types of data and roles.

cache:
  caches:
    - memcached:
        host: <some memcached cluster>
      roles:
        - bloom
        - parquet-footer
    - memcached:
        host: <some memcached cluster>
      roles:
        - frontend-search

Security fixes

This release addresses the following vulnerabilities:

Addressed CVE-2023-5363.
Updated the memcached default image in Jsonnet for multiple CVEs. PR 3310
Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 PR 3613

Bugfixes

For a complete list, refer to the Tempo CHANGELOG.

2.4.2

Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288. PR 3613

2.4.1

Fixed compaction/retention in AWS S3 and GCS when a prefix is configured. PR 3465

2.4.0

Prevent building parquet iterators that would loop forever. PR 3159
Sanitize name in mapped dimensions in span-metrics processor. PR 3171
Fixed an issue where cached footers were requested then ignored. PR 3196
Fixed a panic in autocomplete when the query condition had the wrong type. PR 3277
Fixed TLS when GRPC is enabled on HTTP. PR 3300
Correctly return 400 when max limit is requested on search. PR 3340
Fixed autocomplete filters sometimes returning erroneous results. PR 3339
Fixed trace context propagation between query-frontend and querier. PR 3387
Fixed parsing of span.resource.xyz attributes in TraceQL. PR 3284
Changed exit code if configuration is successfully verified. PR 3174
The tempo-cli analyze blocks command no longer fails on compacted blocks. PR 3183
Moved waitgroup handling for poller error condition. PR 3224
Fixed head block excessive locking in ingester search. PR 3328
Fixed an issue with ingester failed to write traces to disk after a crash or unclean restart. PR 3346

Version 2.3 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.3 release notes

The Tempo team is pleased to announce the release of Tempo 2.3.

This release gives you:

New structural operators in TraceQL:
- Ancestor (<<)
- Parent (<)
- Not-child (!>) (experimental)
- Not-descendant (!>>) (experimental)
- Not-ancestor (!<<) (experimental)
- Not-parent (!<) (experimental)
- Not-siblings (!~) (experimental)
TraceQL support for searching quoted attributes (for example, { span."name with spaces" = "foo" }).
Dedicated attribute columns for improving query performance on your most frequently queried attributes

Tempo 2.3 introduces vParquet3, a Parquet version designed to be more compatible with other Parquet implementations, available as a production-ready option. This block format improves query performance relative to previous formats.

Read the Tempo 2.3 blog post for more examples and details about these improvements.

These release notes highlight the most important features and bugfixes. For a complete list, refer to the Tempo CHANGELOG.

Features and enhancements

The most important features and enhancements in Tempo 2.3 are highlighted below.

vParquet3 promoted to stable

vParquet3, an updated Parquet block format introduced in Tempo 2.2, has been promoted to stable and is production ready. We’re excited about vParquet3 relative to prior formats because of its support for dedicated attribute columns, which help speed up queries on your largest and most queried attributes. [PR 2649]

To read more about the design of vParquet3, refer to the design proposal. For general information, refer to the Apache Parquet schema.

Dedicated attribute columns

Dedicated attribute columns improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list.

Introduced with vParquet3, dedicated attribute columns are only available when using this storage format.

To learn more about this new feature, refer to the Dedicated attribute columns documentation.

TraceQL

Unique to Tempo, TraceQL is a query language that lets you perform custom queries into your tracing data. To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, refer to future work.

We’ve made the following improvements to TraceQL:

Added two structural operators, ancestor (<<) and parent (<) (documentation) [PR 2877]
Added experimental not structural operators: !>>, !>, !<<, !<, and !~ (documentation) [PR 2993]
Added quoted attribute syntax so you can search for attributes with spaces and other special characters in their names (documentation) [PR 3004]
Improved the performance of TraceQL select() queries. Metrics-summary now also correctly handles missing attributes. [PR 2765]
Added support for searching by OpenTelemetry’s span status message using statusMessage intrinsic attribute (documentation) [PR 2848]
Fixed cases where an empty filter ({}) didn’t return expected results [PR 2498]

Metrics-generator

We’ve made the following improvements to metrics-generator:

Added a scope query parameter to /api/overrides so users can choose between fetching the overrides stored by the API and the merged overrides (those actually used by Tempo) [PR 2915, #3018]
Added TempoUserConfigurableOverridesReloadFailing alert [PR 2784]
Added a metrics-generator configuration option to enable/disable X-Scope-OrgID headers on remote write to better support single-tenant setups. [PR 2974]
Allowed metrics-generator ingestion slack to be configured on a per-tenant, rather than global, basis [PR 2589]
Added several metrics-generator fields to user-configurable overrides, including histogram buckets, collection-interval, and span metrics filter policies [PR 2906, 2711, 2928, 2899]

Upgrade considerations

When upgrading to Tempo 2.3, be aware of these considerations and breaking changes.

Transition to vParquet 3

Although the vParquet3 format isn’t yet the default, it’s production ready and we highly recommend switching to it for improved query performance and dedicated attribute columns.

Upgrading to Tempo 2.3 doesn’t modify the Parquet block format. You can use Tempo 2.3 with vParquet2 or vParquet3. vParquet2 remains the default backend for Tempo 2.3; vParquet3 is available as a stable option.

Note
Tempo 2.2 can’t read data stored in vParquet3.

For information on upgrading, refer to Change the block format to vParquet3 upgrade documentation.

Azure SDK v2

If you are using Azure storage, we recommend using the v2 SDK, azure-sdk-for-go. You can use the use_v2_sdk configure option for switching. For more information, refer to the Storage block configuration example documentation. [PR 2952]

Produce debug metrics with the distributor

You can now enable a new configuration block for the distributor to produce debug metrics. These metrics can be particularly useful when tracking down a process that’s creating an exceptional amount of spans. [PR 3008]

distributor:
  metric_received_spans:
    enabled: false
    root_only: false

Changes to the Overrides module configuration

We’ve added a new defaults block to the overrides module for configuring global or per-tenant settings. The Overrides change to indented syntax. For more information, read the Overrides configuration documentation.

You can also use the Tempo CLI to migrate configurations. Refer to the documentation. [PR 2688]

The old configuration block looked like this:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

Here is an example of the new configuration block:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Removed or renamed configuration parameters

Parameter	Comments
`distributor.log_received_traces`	Use the `distributor.log_received_spans` configuration block instead. [PR #3008]
`tempo_query_frontend_queries_total{op="searchtags\|metrics"}`	Removed deprecated frontend metrics configuration option

Security fixes

The following vulnerabilities have been addressed:

Updated Alpine image version to 3.18 to patch CVE-2022-48174 PR 3046
Corrected a HTTP rapid reset issue vulnerability (CVE-2023-39325) PR 3017

Bugfixes

For a complete list, refer to the Tempo CHANGELOG.

2.3.1

Include statusMessage intrinsic attribute in tag search. PR 3084
Fix compactor ignore configured S3 headers. PR 3149
Read session token to s3 credentials. PR 3144

2.3

Loaded defaults for the internal server PR 3041
Fixed pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions PR 3012
Fixed a panic in metrics-summary API PR #2738
Fixed a rare deadlock when uploading blocks to Azure Blob Storage PR 2129
Only search ingester blocks that fall within the request time range. PR 2783
Aligned tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. PR 2840 This query now correctly tells you %age of requests that are within SLO:
traceql
```
sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op)
/
sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
```
Fixed support for blob storage in Azure Stack Hub as a backend. PR 2853
Respected spss on GRPC streaming. PR 2971
Moved empty root span substitution from querier to query-frontend. PR 2671
Ingester errors correctly propagate on the query path PR 2935
Fixed an issue where the ingester didn’t stop a query after timeout PR 3031
Reordered the S3 credential chain and upgraded minio-go. native_aws_auth_enabled is deprecated PR 3006

Version 2.2 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.2 release notes

The Tempo team is pleased to announce the release of Tempo 2.2.

This release gives you:

Major additions to the TraceQL language: structural operators (descendant, child, sibling), results grouped by attribute (by()), a select operator, and 3 new intrinsic attributes.
Faster TraceQL results, thanks to a streaming endpoint that returns partial results as the query executes as well as a multitude of performance-related improvements.
An experimental metrics-summary API that returns RED metrics for recently received spans grouped by your attribute of choice.

Tempo 2.2 makes vParquet2, a Parquet version designed to be more compatible with other Parquet implementations, the default block format. This block format is required for using structural operators and improves query performance relative to previous formats.

Read the Tempo 2.2 blog post for more examples and details about these improvements.

Note
For a complete list of changes, enhancements, and bug fixes refer to the Tempo 2.2 CHANGELOG.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.2 are highlighted below.

Expanding the TraceQL language

With this release, we’ve added to the TraceQL language. TraceQL now offers:

Structural operators: descendant (»), child (>), and sibling (~) (documentation). Find relevant traces based on their structure and relationships among spans. [PR #2625 #2660]
A select() operation that allows you to specify arbitrary span attributes that you want included in the TraceQL response (documentation) [PR 2494]
A by() operation that groups span sets within a trace by an attribute of your choosing. This operation is not supported in the Grafana UI yet; you can only use by() when querying the search API directly. [documentation PR 2490]
New intrinsic attributes for use in TraceQL queries: traceDuration, rootName, and rootServiceName (documentation) [PR #2503]

Read the Tempo 2.2 blog post for examples of how to use these new language additions.

To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, refer to future work.

Get TraceQL results faster

We’re always trying to reduce the time you spend waiting to get results to your TraceQL queries, and we’ve made some nice progress on this front with this release.

We’ve added a gRPC streaming endpoint to the query frontend that allows a client to stream search results from Tempo. The Tempo CLI has been updated to use this new streaming endpoint [PR #2366]. As of version 10.1, Grafana supports it as well, though you must first enable the traceQLStreaming feature toggle [PR #72288].

By streaming results to the client, you can start to look at traces matching your query before the entire query completes. This is particularly helpful for long-running queries; while the total time to complete the query is the same, you can start looking at your first matches before the full set of matched traces is returned.

In addition to streaming partial results, we’ve merged a number of improvements to speed up TraceQL queries. Here are just a few:

Add support for query batching between frontend and queriers to improve throughput [PR 2677]
Improve performance of TraceQL regular expressions [PR 2484]
Fully skip over Parquet row groups with no matches in the column dictionaries [PR 2676]

New synchronous read mode for vParquet and vParquet2 [PRs 2165, 2535]

Improved TraceQL throughput by asynchronously creating jobs. [PR 2530]

Metrics summary API (experimental)

Tempo has added an experimental API that returns RED metrics (span count, erroring span count, and latency information) for spans of kind=server sent in the last hour, grouped by an attribute of your choice. For example, you could use this API to compare error rates of spans with different values of the namespace attribute. From here, you might see that spans from namespace=A have a significantly higher error rate than those from namespace=B. As another example, you could use this API to compare latencies of your spans broken down by the region attribute. From here, you might notice that spans from region=North-America have higher latencies than those from region=Asia-Pacific.

This API is meant to enable as-needed analysis of your incoming spans; by segmenting your spans by attribute and looking for differences in RED metrics, you can more quickly isolate where problems like elevated error rates or higher latencies are coming from.

Unlike RED metrics computed by the metrics-generator, the values returned by this API aren’t persisted as time series. This has the advantage that you don’t need to provide your own time series databases for storing and querying these metrics. It also allows you to compute RED metrics broken down by high cardinality attributes that would be too expensive to store in a time series database. Use the metrics generator if you want to store and visualize RED metrics over multi-hour or multi-day time ranges, or you want to alert on these metrics.

To learn more about this API, refer to the metrics summary API documentation.

This work is represented in multiple PRs: 2368, 2418, 2424, 2442, 2480, 2481, 2501, 2579, and 2582.

Other enhancements

The tag values and tag names APIs now support filtering [PR 2253]. This lets you retrieve all valid attribute values and names given certain criteria. For example, you can get a list of values for the attribute namespace seen on spans with attribute resource=A. This feature is off by default; to enable, configure autocomplete_filtering_enabled. ( documentation). The autocomplete in Grafana can make use of this filtering capability to provide better suggestions starting in v10.2 PR [67845].
The metrics-generator now supports span filtering. Setting up filters allows you to compute metrics over the specific spans you care about, excluding others. It also can reduce the cardinality of generated metrics, and therefore the cost of storing those metrics in a Prometheus-compatible TSDB. (documentation) [PR 2274]
The metrics-generator can now detect virtual nodes ( documentation) [PR 2365]. As a result, you’ll now see these virtual nodes represented in your service graph.

Upgrade considerations

When upgrading to Tempo 2.2, be aware of these breaking changes:

JSonnet users only: We’ve converted the metrics-generator component from a k8s deployment to a k8s statefulset. Refer to the PR for seamless migration instructions. [PRs #2533, #2467]
Removed or renamed configuration parameters (see section below)

While not a breaking change, upgrading to Tempo 2.2 will by default change the block format to vParquet2. To stay on a previous block format, read the Parquet configuration documentation. We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the duration intrinsic.

Removed or renamed configuration parameters

The following fields were removed or renamed.

storage:

trace:

s3:

insecure_skip_verify: true // renamed to tls_insecure_skip_verify

Parameter Comments

query_frontend:

tolerate_failed_blocks: <int>

Remove support for `tolerant_failed_blocks` [PR 2416]

Renamed `insecure_skip_verify` to `tls_insecure_skip_verify` [PR 2407]

Bug fixes

For a complete list, refer to the Tempo CHANGELOG.

2.2.4

Updated Alpine image version to 3.18 to patch CVE-2022-48174 PR 3046
Bumped Jaeger query docker image to 1.50.0 PR 2998

2.2.3

Fixed S3 credentials providers configuration PR 2889

2.2.2

Fixed node role auth IDMSv1 PR 2760

2.2.1

Fixed incorrect metrics for index failures PR 2781
Fixed a panic in the metrics-generator when using multiple tenants with default overrides PR 2786
Restored tenant_header_key removed in PR 2414 PR 2786
Disabled streaming over HTTP by default PR 2803

2.2

Fixed an issue in the metrics-generator that prevented scaling up parallelism when remote writing of metrics was lagging behind [PR 2463]
Fixed an issue where metrics-generator was setting wrong labels for traces_target_info [PR 2546]
Fixed an issue where matches and other spanset level attributes were not persisted to the TraceQL results. [PR 2490]
Fixed an issue where ingester search could occasionally fail with file does not exist error [PR 2534]

Version 2.1 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.1 release notes

The Tempo team is pleased to announce the release of Tempo 2.1.

With this release, Tempo enhances TraceQL so you can select traces using features such as span kind, aribtrary arithmetic, and new aggregates min/max/sum. In addition, Tempo 2.1 introduces vParquet2, a new experimental Parquet version designed to be more compatible with other Parquet implementations.

Note: For a complete list of changes and enhancements, refer to the Tempo 2.1 changelog.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.1 are highlighted below.

Enhancements to TraceQL

With this release, we’ve expanded the TraceQL language. In Tempo 2.0, TraceQL allowed users to select traces based upon resource attributes, time, duration, and basic aggregates. In Tempo 2.1, users can also select traces based on the following:

Enable querying by kind, an intrinsic attribute (PR 2217)
- { kind = server }, { kind = client }, { kind = consumer }
- { kind = server && duration > 1s }
Support arithmetic expressions (PR 2146)
- { span.bytesProcessed > 10 * 1024 * 1024 }
- { span.bytesProcessed / 1024 / 1024 > 10 }
- { span.jobs / span.bytes > 3 }
Enable min, max, and sum aggregate filters
- {} | min(duration) > 1s
- {} | max(span.bytesProcessed) < 1024 * 1024
- {} | sum(duration) > 1h
Support for more numeric types: Compare floating point numbers and integers, allow floating point numbers for durations, and support negative numbers when doing aggregate filters. (PR 2139, 2304, 2289)
- { 1 = 1.0 }
- { duration > 1.5s }, {duration > 1s500ms }
- {} | count() > -1

This update also includes major performance improvements to TraceQL query performance. [PR 2119]

To learn more about the TraceQL syntax, see the TraceQL documentation. For information on planned future extensions to the TraceQL language, see future work.

TraceQL query editor improvements (experimental)

Starting in Grafana 9.5, users have access to a Tempo Search tab powered by TraceQL. To enable this feature, use Grafana’s traceqlSearch feature flag. The editor can autocomplete queries and has a streamlined interface.

Grafana will soon include a new Search screen that can build a TraceQL query based upon options selected in drop-downs. Contact Support to enable this feature in Grafana Cloud.

For more information, refer to the TraceQL query editor documentation.

vParquet2 (experimental)

Tempo 2.1 offers an experimental update to the Parquet backend: vParquet2. Due to the dedicated span duration column, there is a performance gain for TraceQL queries containing duration. vParquet2 will also be a prerequisite for TraceQL’s structural operators, which will be introduced in a future release. [PR 2244]

You can select the block format Tempo uses by changing the block format version option. To use the new format, set the block format to vParquet2. For more information, read the Parquet configuration documentation.

The new encoding implements Parquet schema changes that are not compatible with vParquet:

Renamed columns to make the schema more consistent and closer to names and terminologies used by OTEL
Add a dedicated span column for duration
Additional columns that will enable new TraceQL features, including hierarchical queries

Other enhancements and features

For a complete list, refer to the Tempo 2.1 changelog.

PR 2105 Log when a trace is too large to compact. Traces are dropped when they exceed a certain limit set in the ingesters. Any dropped traces are now logged with the number of dropped spans and the trace ID.
PR 2195 Add support for Azure Workload Identity authentication
PR 2130 Add command to tempo-cli to migrate a tenant
PR 2093 Add support for s3 session token in static configuration

Upgrade considerations

When upgrading to Tempo 2.1, be aware of these breaking changes.

Breaking changes to metric names exposed by Tempo

All Prometheus metrics exposed by Tempo on its /metrics endpoint that were previously prefixed with cortex_ have now been renamed to be prefixed with tempo_ instead. (PR 2204)

Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR 2008)

The query_frontend_result_metrics_inspected_bytes metric was removed in favor of query_frontend_bytes_processed_per_second.

Remove support for Search on v2 blocks

Users can no longer search blocks in v2 format. Only vParquet and vParquet2 formats support search. The following search configuration options were removed from the overrides section:

overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:

The following metrics configuration was also removed:

tempo_ingester_trace_search_bytes_discarded_total

This change means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. One way to avoid this is to upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces. Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable.

Finally, parquet files are no longer cached when carrying out searches (PR 2164).

Bug fixes

For a complete list, refer to the Tempo 2.1 changelog.

2.1.1

PR 2400 Fixed issue where Tempo sometimes flips booleans from false->true at storage time.

2.1

PR 2220 Correctly connect context during compaction
PR 2081 Apply rate() to the bytes/s panel in the tenant’s dashboard.
PR 2111 Retry copy operations during compaction in GCS backend.
PR 2164 Improve locking and search head block in SearchTagValuesV2
PR 2152 Fix not closing WAL block file before attempting to delete the folder.
PR 2219 Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like status.
PR 2254 Correctly return unique spans when &&ing and ||ing spansets.

Version 2.0 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 2.0 release notes

The Tempo team is pleased to announce the newest major release of Tempo: version 2.0!

Tempo 2.0’s headline feature is the introduction of TraceQL. TraceQL is a query language modeled on PromQL and LogQL that allows you to interactively extract insights from your trace data. It provides a powerful, flexible way to pinpoint the traces you need to answer questions about your systems and analyze traces based on their structure.

Continuing in the tradition of Tempo’s close integration with Grafana as a visualization layer, we’ve also added a TraceQL query editor to Grafana’s Tempo data source. The editor is available in Grafana Cloud and for open source users in Grafana 9.3.2. We encourage open source users to upgrade to Grafana 9.4 as soon as it is released to take advantage of some additional enhancements and bug fixes.

In addition to TraceQL, Tempo 2.0 also marks the promotion of the Apache Parquet block format to the new default storage format. With this change, the Tempo maintainers are signaling that this new format has received enough testing to be ready for broad adoption.

These two changes actually go hand-in-hand: without the increased search speed enabled by the Apache Parquet block format, it would be impossible to pull off the sophisticated queries made possible by TraceQL.

Note: For a complete list of changes and enhancements, refer to the Tempo 2.0 changelog.

Features and enhancements

Some of the most important features and enhancements in Tempo 2.0 are highlighted below.

TraceQL, the trace query language

Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. TraceQL makes it possible to find traces that otherwise would have been difficult or even impossible to identify with Tempo’s existing search. It also makes it possible for you to understand the broad context for an event sequence. To understand more about why we built TraceQL and the use cases we think it unlocks, see our blog post.

With this release, we’ve implemented a subset of the full TraceQL language we envision. Users can select traces based on the following:

Span and resource attributes, timing, and duration
Basic aggregates: count() and avg()

To learn more about the TraceQL syntax, see the TraceQL documentation.

You can run a TraceQL query either by issuing it to Tempo’s q parameter of the search API endpoint, or, for those using Tempo in conjunction with Grafana, by using Grafana’s TraceQL query editor, described below.

TraceQL requires Tempo’s Parquet columnar format, which is the default block format in Tempo 2.0. For more information, refer to the Tempo Apache Parquet block format documentation.

TraceQL query editor in Grafana

To make it easier to run and visualize the results of TraceQL queries, we’ve added a TraceQL query editor to Grafana’s Tempo data source. The editor is available in Grafana’s Explore interface starting in Grafana version 9.3.2.

The query editor helps you learn TraceQL by offering autocomplete.

For more information, refer to the TraceQL query editor documentation.

Apache Parquet block format is the new default

The experimental Apache Parquet columnar block format introduced in Tempo v1.5 has been promoted to stable and is now the default trace storage for Tempo 2.0. We’re excited to switch to it because of the increased speed and efficiency it provides for search.

With our previous block format, we’d seen our ability to search trace data cap out at a rate of ~40-50 GB per second. Since switching to Parquet, we’re now hitting search speeds of 300 GB/s on common queries and doing so with less compute. If we define the bytes processed as the total trace bytes that are scanned, then this example query resulted in ~1.2 TB/s:

{ resource.cluster ="foo" }

and this query resulted in ~550 GB/s:

{ span.foo ="bar" }

Another benefit of the Parquet block format relative to our previous format is that users can now leverage the large ecosystem of existing Parquet tools and libraries to process and transform their trace data. We look forward to see what downstream applications people create with this ability.

The Parquet block format can be used as a drop-in replacement for Tempo’s existing block format. No data conversion or upgrade process is necessary. As soon as the Parquet format is enabled, Tempo starts writing data in that format, leaving existing data as-is.

Tempo search will return results from both the legacy and Parquet block formats. However, search on legacy blocks has been marked as deprecated and will be removed in Tempo 2.1.

TraceQL queries will only search trace data stored in Parquet blocks. For more information, refer to the Parquet documentation.

Note: There is a potential issue loading Tempo 1.5’s experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues please report them so we can isolate and fix this issue.

Search and metrics-generator enabled by default

Search and metrics-generator are now enabled by default in Tempo 2.0, and their corresponding configuration parameters (search_enabled and metrics_generator_enabled) have been removed.

We’ve been excited by the feedback we’ve gotten on both features and are satisfied with their stability so we want all newcomers to Tempo to have access to them from day 1.

Although the metrics-generator component is now enabled by default, no actual metrics will be generated until certain tenant-specific configuration parameters have been set. For more information, refer to the metrics-generator configuration documentation.

NOTE: Grafana Cloud customers need to contact Grafana Support to enable metrics generation.

As part of promoting the metrics-generator to default on, we’ve made the following improvements to it:

The metrics-generator now handles collisions between user-defined and default dimensions. User-defined dimensions colliding with default dimensions will be prefixed with __. PR 1794
The metrics-generator now makes intrinsic dimensions configurable and disables status_message by default PR 1960.
The metrics-generator now computes a metric that tells you the size (in bytes) of spans received broken down by team, service, or other relevant labels. Since span volume roughly correlates with the cost to operate Tempo, this metric is helpful for cost attribution and chargeback purposes. PR 1662.
Older spans are now filtered out before metrics are aggregated. This is helpful since including these older span in metric computation can lead to misleading or incorrect results. PR 1612.
Label names and values that exceed a configurable length are truncated. In this way, the Tempo metrics-generator sanitizes metrics before sending them to a receiving time-series-database (which usually has its own limits around maximum label length). PR 1897.

Zone-aware replication for ingesters

Zone awareness is a feature that ensures data is replicated across failure domains (which we refer to as “zones”) to provide greater reliability. A failure domain is whatever you define it to be, but commonly may be an availability zone, data center, or server rack.

When zone awareness is enabled for ingesters, incoming trace data is guaranteed to be replicated to ingesters in different zones. This allows the system to withstand the loss of one or more zones (depending on the replication factor).

Thank you to community member manohar-koukuntla for making this contribution!

# use the following fields in _config field of jsonnet config, to enable zone aware ingesters.
multi_zone_ingester_enabled: false,
multi_zone_ingester_migration_enabled: false,
multi_zone_ingester_replicas: 0,
multi_zone_ingester_max_unavailable: 25,

Upgrade considerations

When upgrading to Tempo 2.0, be aware of these breaking changes.

Defaults updated

The following configuration parameters have been updated with new default values. These defaults were changed to work better with the new Parquet block format. PR 1978

If you are sticking with the v2 format you may prefer the old values.

query_frontend:
  max_oustanding_per_tenant: 2000
  search:
    concurrent_jobs: 1000
    target_bytes_per_job: 104857600
    max_duration: 168h
    query_ingesters_until: 30m
  trace_by_id:
    query_shards: 50
querier:
  max_concurrent_queries: 20
  search:
    prefer_self: 10
ingester:
  concurrent_flushes: 4
  max_block_duration: 30m
  max_block_bytes: 524288000
storage:
  trace:
    pool:
      max_workers: 400
      queue_depth: 20000
    search:
      read_buffer_count: 32
      read_buffer_size_bytes: 1048576

Removed and renamed configuration parameters

The following tables describe the parameters that have been removed or renamed. (PR#1978)

Removed

Parameter	Comments
query_frontend: query_shards:	Replaced by `trace_by_id.query_shards`
querier: query_timeout:	Replaced by two different settings: `search.query_timeout` and `trace_by_id.query_timeout`
ingester: use_flatbuffer_search:	Removed and automatically determined based on block format
`search_enabled`	Removed. Now defaults to true.
`metrics_generator_enabled`	Removed. Now defaults to true.
storage: wal: version:	Removed and pinned to `block.version`

Renamed

Parameter	Comments
query_frontend: query_shards:	Renamed `query_frontend.trace_by_id.query_shards`
querier: query_timeout:	Renamed `querier.trace_by_id.query_timeout`

The following compactor configuration parameters were renamed.

Parameter	Comments
compaction: chunk_size_bytes:	Renamed to `v2_in_buffer_bytes`
compaction: flush_size_bytes:	Renamed to `v2_out_buffer_bytes`
compaction: iterator_buffer_size:	Renamed to `v2_prefetch_traces_count`

The following storage configuration parameters were renamed.

Parameter	Comments
wal: encoding:	Renamed to `v2_encoding`
block: index_downsample_bytes:	Renamed to `v2_index_downsample_bytes`
block: index_page_size_bytes:	Renamed to `v2_index_page_size_bytes`
block: encoding:	Renamed to `v2_encoding`
block: row_group_size_bytes:	Renamed to `parquet_row_group_size_bytes`

Other upgrade considerations

PR 1879. The Azure Storage configuration now uses snake case with underscores (_) instead of dashes (-)Use snake case on Azure Storage config. Here is an example of using snake case on Azure Storage config:

# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:

PR 1678. Parquet is the new default block version. To continue using the v2 block format, see the Parquet configuration documentation.

PR 1810. We’ve deleted the TempoRequestErrors alert from mixin. Any Jsonnet users relying on this alert should copy this into their own environment.

PR 1754. We’ve added TLS support to the vulture. Internal types are updated to use scope instead of instrumentation_library. This is a breaking change in “trace by ID” type queries if JSON is requested.

Bug fixes

2.0.1

Enhancements

PR 2059 No longer return status.code from /api/search/tags unless it is an attribute present in the data,
PR 2062 Pad leading zeroes in span id to always be 16 characters.

Bug fixes

PR 2058 Suppress logspam in single binary mode when metrics generator is disabled.
PR 2055 Error more gracefully while reading some blocks written by an interim commit between 1.5 and 2.0.
PR 2095 Correctly coalesce trace level data when combining Parquet traces.
PR 2114 Unescape query parameters in AWS Lambda to allow TraceQL queries to work.

2.0 bug fixes

Version 2.0 includes the following fixes:

PR 1887 Stop distributors on OTel receiver fatal error.
PR 1700 New wal file separator + for the NTFS filesystem and backward compatibility with the old separator :.
PR 1697 Honor caching and buffering settings when finding traces by ID.
PR 1723 Correctly propagate errors from the iterator layer up through the queriers.
PR 1781 Make multitenancy work with HTTP.
PR 1799 Fix parquet search bug fix on http.status_code that may cause incorrect results to be returned.
PR 1813 Fix failing SearchTagValues endpoint after startup.
PR 1913 tempo-mixin: tweak dashboards to support metrics without cluster label present.
PR 1920 Fix docker-compose examples not running on Apple M1 hardware.
PR 1939 Fix TraceQL parsing of most binary operations to not require spacing.
PR 1947 Don’t persist tenants without blocks in the ingester.
PR 1948 TraceQL: span scope not working with ranges.
PR 1997 TraceQL: skip live traces search.
PR 2003 Return more consistent search results by combining partial traces.

Version 1 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 1 release notes

Release notes for Grafana Tempo version 1.x releases.

Release notes on Grafana Labs

Version 2.10 release notes

Version 2.10 release notes

TraceQL

Find traces with missing attributes (= nil)

Analyze trace structure with span:childCount

Reference integer boundaries with minInt and maxInt

Metrics-generator

Additional metrics-generator improvements

vParquet5

Additional vParquet5 updates

LLM-optimized API responses

Features and enhancements

TraceQL correctness

Performance improvements

User-configurable overrides

Other enhancements

Upgrade considerations

Search default result limit changed in 2.10.2

Busybox removed from Tempo image

vParquet format changes

Enable vParquet5 (optional)

vParquet2 removed

vParquet3 and vParquet2 schemas deprecated

Breaking changes

Project Rhythm: New Tempo architecture

Tempo rearchitecture (experimental)

Security fixes

2.10.3

2.10.2

Bug fixes

2.10.2

2.10.1

2.10.0

Version 2.9 release notes

Version 2.9 release notes

Access tracing data with the Tempo MCP server

TraceQL metrics sampling

Features and enhancements

Operational improvements

SLO metrics improvements

Enhanced monitoring and observability

Ingress bytes monitoring

Metrics-generator improvements

TraceQL correctness and performance improvements

Performance improvements

Upgrade considerations

RPM and DEB packages discontinued

Migrated Vulture and integration tests to OTLP exporter

Bucket calculation changes

Series label handling improvements

vParquet5 block format

Go version upgrade

Project Rhythm: New Tempo architecture

Current problems

Architecture comparison

Security fixes

2.9.1

Bug fixes

Version 2.8 release notes

Version 2.8 release notes

New language features for TraceQL

Ranking topk and bottomk functions for TraceQL metrics

The sum_over_time function for TraceQL metrics

Experimental query hint most_recent=true to retrieve the most recent traces

Query by parent span ID

Major performance and memory usage improvements

Performance improvements

Features and enhancements

TraceQL correctness

TraceQL exemplar improvements and bugfixes

Improved error handling and logging

Other enhancements and improvements

Upgrade considerations

Changed the default listening port

Removed Tempo serverless

Updated, removed, or renamed configuration parameters

Other upgrade considerations

Security fixes

2.8.3

Find traces with missing attributes (`= nil`)

Analyze trace structure with `span:childCount`

Reference integer boundaries with `minInt` and `maxInt`

Ranking `topk` and `bottomk` functions for TraceQL metrics

The `sum_over_time` function for TraceQL metrics

Experimental query hint `most_recent=true` to retrieve the most recent traces

OpenTelemetry Collector receiver listens on `localhost` by default