--- title: "Version 2.8 release notes | Grafana Tempo documentation" description: "Release notes for Grafana Tempo 2.8" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Version 2.8 release notes The Tempo team is pleased to announce the release of Tempo 2.8. This release gives you: - New TraceQL features such as `most_recent`, parent-span filters, `sum_over_time`, and `topk`/`bottomk` to help surface the exact spans and metrics you need. - Concurrent block flushing, iterator optimizations, and back-pressure make ingestion and queries leaner and quicker. - Safer defaults with port 3200, distroless images, Go 1.24, and an upgraded OTLP Collector tighten security and simplify ops. These release notes highlight the most important features and bugfixes. For a complete list, refer to the [Tempo changelog](https://github.com/grafana/tempo/releases). > Note > > Tempo 2.8 has some major upgrade considerations including changing the default `http-listen-port` from 80 to 3200, removing serverless, and enforcing max attribute size at event, link, and instrumentation scopes. > > Refer to [Upgrade considerations](#upgrade-considerations) for details. Read the [Tempo 2.8 blog post](/blog/2025/06/12/grafana-tempo-2.8-release-memory-improvements-new-traceql-features-and-more/) for more examples and details about these improvements. ## New language features for TraceQL We’re excited to roll out three powerful enhancements to TraceQL, giving you more flexibility and performance when querying traces in Grafana Cloud with Tempo. - Rank your metrics with new `topk(n)` and `bottomk(n)` functions to quickly get your highest and lowest ranking time series. - Aggregate spans over time using `sum_over_time()` for built-in cumulative sums, such as total bytes, error counts. - Fetch the latest traces first via the experimental `most_recent=true` query hint. ### Ranking `topk` and `bottomk` functions for TraceQL metrics When you’re looking at latency, error rates, or throughput across hundreds or thousands of services or endpoints, it’s easy to get lost in all the data. Previously, you’d have to pull back the full set of aggregates and then manually inspect or post‑process the results to find your worst offenders or best performers. With `topk(n)` and `bottomk(n)`, you can immediately narrow your focus to the top‑ or bottom‑ranked spans in a single, efficient query. This saves time and reduces the data volume you need to scan downstream. ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none {} | avg_over_time(span:duration) by (span:name) | topk(10) {} | count_over_time() by (span:name) | bottomk(10) ``` These are second stage functions used, where: - `topk(n)` returns the *n* series with the highest values from a first‑stage aggregation. - `bottomk(n)` returns the *n* series with the lowest values. For more information, refer to the [topk and bottomk documentation](/docs/tempo/next/traceql/metrics-queries/functions/#topk-and-bottomk-functions) ([PR 4646](https://github.com/grafana/tempo/pull/4646/)). ### The `sum_over_time` function for TraceQL metrics The `sum_over_time()` function lets you aggregate numerical values by computing the sum value of them. The time interval that the sum is computed over is set by the `step` parameter. ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none { span.http.response_content_length > 0 } | sum_over_time(span.http.response_content_length) ``` With `sum_over_time()`, you can directly compute cumulative sums inside TraceQL, like total bytes transferred, total error counts, or resource consumption over time. ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none {} | sum_over_time(span.http.response_content_length) ``` For each step interval, `sum_over_time(attr)` totals the values of `attr` across all matching spans. Refer to [TraceQL metrics functions](/docs/tempo/next/traceql/metrics-queries/functions/#the-sum_over_time-min_over_time-max_over_time-and-avg_over_time--functions) for more information. ([PR 4786](https://github.com/grafana/tempo/pull/4786)) ### Experimental query hint `most_recent=true` to retrieve the most recent traces When troubleshooting a live incident or monitoring production health, you often need to see the absolute latest traces first. By default, Tempo’s query engine favors speed and returns the first `N` matching traces, which may not be the newest. The `most_recent` hint ensures you’re always looking at the freshest data, so you can diagnose recent errors or performance regressions without missing anything due to early row‑limit cuts. Examples: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none {} with (most_recent=true) { span.foo = "bar" } >> { status = error } with (most_recent=true) ``` With `most_recent=true`, Tempo performs a deeper search across data shards, retains the newest candidates, and returns traces sorted by start time rather than stopping at the first limit hit. ([PR 4238](https://github.com/grafana/tempo/pull/4238)) The TraceQL query hint `most_recent=true` can be used with any TraceQL selection query to force Tempo to return the most recent results ordered by time. For more information, refer to [Retrieve most recent results (experimental)](/docs/tempo/next/traceql/construct-traceql-queries/#retrieve-most-recent-results-experimental). ### Query by parent span ID In addition, you can query by parent `span id`. Using `span:parentID`, you can ensure that you are looking at a child span of a specific parent, which is useful for example in span linking. Refer to the [documentation](/docs/tempo/next/traceql/construct-traceql-queries/#intrinsic-fields) for more information. ([PR 4692](https://github.com/grafana/tempo/pull/4692)) ## Major performance and memory usage improvements Tempo 2.8 removes pooling for large traces. In testing, removing large pooling impacted memory, cutting memory high-water marks to less than half in a high traffic/tenant installation. Some amount of pooling is needed to keep CPU, throughput, and GC rate maintained. Now, we have a fixed 1M default allocation size, which covers most traces. This graph illustrate the amount of memory used by the compactors. In one cell, we saw major improvements in compactor memory usage after PR [4985](https://github.com/grafana/tempo/pull/4985) was merged. Refer to PR [4985](https://github.com/grafana/tempo/pull/4985) for more information, including benchmarks and test results. [](/media/docs/tempo/tempo-2.8-pr-4985-compactor-memory-graph.png) ### Performance improvements TraceQL and Tempo generally see a number of performance improvements in this release. Refer to benchmarks in the PRs for additional information. - Improve TraceQL performance by reverting EqualRowNumber to an inlineable function. ([PR 4705](https://github.com/grafana/tempo/pull/4705)) - Improve iterator performance by using max definition level to ignore parts of the RowNumber while nexting. Nexting is a function multiple in Tempo when you do a big query. Even minor improvements in nexting can have a major impact. ([PR 4753](https://github.com/grafana/tempo/pull/4753)) - Increase query-frontend default batch size. TraceQL performance has improved enough to pass larger batches than before and still complete all jobs on one querier. ([PR 4844](https://github.com/grafana/tempo/pull/4844)) - Improve Tempo build options. ([PR 4755](https://github.com/grafana/tempo/pull/4755)) - Rewrite traces using rebatching. ([PR 4690](https://github.com/grafana/tempo/pull/4690)) - Reorder span iterators. ([PR 4754](https://github.com/grafana/tempo/pull/4754)) - Improve memcached memory usage by pooling buffers. ([PR 4970](https://github.com/grafana/tempo/pull/4970)) ## Features and enhancements This section highlights the most important features and enhancements in Tempo 2.8. ### TraceQL correctness One of our major focuses with this release has been to improve TraceQL’s performance and correctness. The following list summarizes some of the more important improvements. - Fixed behavior for queries like `{.foo && true}` and `{.foo || false}`. ([PR 4855](https://github.com/grafana/tempo/pull/4855)) - Make comparison to nil symmetric. ([PR 4869](https://github.com/grafana/tempo/pull/4869)) - Corrected TraceQL incorrect results for additional spanset filters after a select operation. ([PR 4600](https://github.com/grafana/tempo/pull/4600)) - Fixed TraceQL metrics incorrect results for queries with multiple filters that reside in non-dedicated columns that also group by the same variable. ([PR 4887](https://github.com/grafana/tempo/pull/4887)) - Fixed metrics streaming for all non-trivial metrics. ([PR 4624](https://github.com/grafana/tempo/pull/4624)) - Fixed behavior of `{} >> { span.attr-that-doesnt-exist != "foo" }`. ([PR 5007](https://github.com/grafana/tempo/pull/5007)) - Fixed various edge cases for query range. ([PR 4962](https://github.com/grafana/tempo/pull/4962)) - Excluded `nestedSetParent` and other values from the `compare()` function. ([PR 5196](https://github.com/grafana/tempo/pull/5196)) The query frontend can cache job results for individual blocks when executing TraceQL. These bug fixes impact caching behavior. - TraceQL results caching bug for floats ending in `.0`. ([PR 4539](https://github.com/grafana/tempo/pull/4539)) - Correctly cache frontend jobs for query range in TraceQL metrics. ([PR 4771](https://github.com/grafana/tempo/pull/4771)) - Fixed frontend cache key generation for TraceQL metrics queries to prevent collisions. ([PR 5017](https://github.com/grafana/tempo/pull/5017)) ### TraceQL exemplar improvements and bugfixes Tempo 2.8 improves exemplars in TraceQL metrics. - Distribute exemplars over time and hard limit the number of exemplars. ([PR 5158](https://github.com/grafana/tempo/pull/5158)) - Right exemplars for histogram and quantiles. ([PR 5145](https://github.com/grafana/tempo/pull/5145)) - Fixed for queried number of exemplars. ([PR 5115](https://github.com/grafana/tempo/pull/5115)) ### Improved error handling and logging In addition, we’ve updated how Tempo handles errors so it’s easier to troubleshoot. - Updated query range error message. ([PR 4929](https://github.com/grafana/tempo/pull/4929)) - Improved rate limit error message when traces size exceeds rate limit. ([PR 4986](https://github.com/grafana/tempo/pull/4986/)) - Restored dedicated columns logging for completed blocks in the compactor. ([PR 4832](https://github.com/grafana/tempo/pull/4832)) - Query-frontend logs add a message to the log line. ([PR 4975](https://github.com/grafana/tempo/pull/4975)) ### Other enhancements and improvements This release also has these notable updates. - Added throughput SLO and metrics for the `TraceByID` endpoint. Configurable via the `throughput_bytes_slo` field, and it will populate `op="traces"` label in SLO and throughput metrics. ([PR 4668](https://github.com/grafana/tempo/pull/4668)) - Prevented queries in the ingester from blocking flushing traces to disk and memory spikes. ([PR 4483](https://github.com/grafana/tempo/pull/4483)) - Host Info Processor now tracks host identifying resource attribute in metric ([PR 5152](https://github.com/grafana/tempo/pull/5152)) - Added IPv6 support to the distributor. ([PR 4840](https://github.com/grafana/tempo/pull/4840)) - Added default mutex and blocking values. ([PR 4979](https://github.com/grafana/tempo/pull/4979)) ## Upgrade considerations When [upgrading](/docs/tempo/next/set-up-for-tracing/setup-tempo/upgrade/) to Tempo 2.8, be aware of these considerations and breaking changes. ### Changed the default listening port With Tempo 2.8, the default `http_listen_port` changes from 80 to 3200. Check the configuration options for the `server:` block in your Tempo configuration file. YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml server: # HTTP server listen host [http_listen_address: ] # HTTP server listen port [http_listen_port: | default = 3200] ``` Refer to [issue 4945](https://github.com/grafana/tempo/discussions/4945) for more information for the rationale. [PR 4960](https://github.com/grafana/tempo/pull/4960) ### Removed Tempo serverless Tempo serverless has been removed. The following configuration options are no longer valid and should be removed from your Tempo configuration. ([PR 4599](https://github.com/grafana/tempo/pull/4599/)) YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml querier: search: prefer_self: external_hedge_requests_at: external_hedge_requests_up_to: external_backend: google_cloud_run: external_endpoints: ``` In addition, these Tempo serverless related metrics have been removed: `tempo_querier_external_endpoint_duration_seconds`, `tempo_querier_external_endpoint_hedged_roundtrips_total`, and `tempo_feature_enabled`. ### Updated, removed, or renamed configuration parameters Expand table | Parameter | Comments | |-------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `max_span_attr_byte` | Renamed to `max_attribute_bytes`. ([PR 4633](https://github.com/grafana/tempo/pull/4633)) | | `tempo_discarded_spans_total` | Removed `internal_error` as a reason from `tempo_discarded_spans_total`. ([PR 4554](https://github.com/grafana/tempo/pull/4554)) | | `tempo_receiver_accepted_span` and `tempo_receiver_refused_spans` | The `name` dimension from `tempo_receiver_accepted_span` and `tempo_receiver_refused_spans` changes from `tempo/jaeger_receiver` to `jaeger/jaeger_receiver`. ([PR 4893](https://github.com/grafana/tempo/pull/4893)) | ### Other upgrade considerations - Upgrade OTEL Collector to v0.122.1. The `name` dimension from `tempo_receiver_accepted_span` and `tempo_receiver_refused_spans` changes from `tempo/jaeger_receiver` to `jaeger/jaeger_receiver`. ([PR 4893](https://github.com/grafana/tempo/pull/4893)) - Replace `opentracing-contrib/go-grpc` by `otelgrpc` in Tempo query. ([PR 4958](https://github.com/grafana/tempo/pull/4958)) - Enforce max attribute size at event, link, and instrumentation scope. The configuration is now per-tenant. Renamed `max_span_attr_byte` to `max_attribute_bytes`. ([PR 4633](https://github.com/grafana/tempo/pull/4633)) - Converted SLO metric `query_frontend_bytes_processed_per_second` from a histogram to a counter as it’s more performant. ([PR 4748](https://github.com/grafana/tempo/pull/4748)) - Removed the OpenTelemetry Jaeger exporter, which has been [deprecated](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/jaeger). ([PR 4926](https://github.com/grafana/tempo/pull/4926)) ## Security fixes The following updates were made to address security issues. ### 2.8.4 - Updated Go to version 1.26.2 to address [CVE-2026-25679](https://nvd.nist.gov/vuln/detail/CVE-2026-25679). \[[PR 6800](https://github.com/grafana/tempo/pull/6800)] - Patched gRPC modules to v1.79.3 and upgraded `go.opentelemetry.io/otel/sdk` to v1.40.0 to address [CVE-2026-33186](https://github.com/advisories/GHSA-p77j-4mvh-x3m3) and [CVE-2026-39883](https://github.com/advisories/GHSA-hfvc-g4fc-pqhx). \[[PR 6899](https://github.com/grafana/tempo/pull/6899)] - Updated `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` to v1.43.0 to address [CVE-2026-39882](https://github.com/advisories/GHSA-w8rr-5gcm-pp58). \[[PR 6890](https://github.com/grafana/tempo/pull/6890)] - Updated `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp` to v1.43.0 to address [CVE-2026-39882](https://github.com/advisories/GHSA-w8rr-5gcm-pp58). \[[PR 6889](https://github.com/grafana/tempo/pull/6889)] - Updated `github.com/go-jose/go-jose/v4` to v4.1.4 to address [CVE-2026-34986](https://github.com/advisories/GHSA-78h2-9frx-2jm8). \[[PR 6908](https://github.com/grafana/tempo/pull/6908)] - Updated `github.com/antchfx/xpath` to v1.3.6 to address [CVE-2026-32287](https://nvd.nist.gov/vuln/detail/CVE-2026-32287). \[[PR 6763](https://github.com/grafana/tempo/pull/6763)] - Capped the exemplars hint in TraceQL metrics queries to prevent unbounded memory allocation. Addresses CVE-2026-27878. \[[PR 6792](https://github.com/grafana/tempo/pull/6792)] - Set default `max_result_limit` for search to `262144` (256×1024). The previous default of `0` allowed unbounded search result sets. Addresses CVE-2026-21728. \[[PR 6525](https://github.com/grafana/tempo/pull/6525)] ### 2.8.3 - Updated Go to version 1.25.5 to address [CVE-2025-61729](https://github.com/advisories/GHSA-7c64-f9jr-v9h2), [CVE-2025-47907](https://github.com/advisories/GHSA-j5pm-7495-qmr3), [CVE-2025-58183](https://github.com/advisories/GHSA-9gcr-gp5f-jw27), and [CVE-2025-61727](https://github.com/advisories/GHSA-5mh9-3jwc-rp59). \[[PRs 6089](https://github.com/grafana/tempo/pull/6089), [6096](https://github.com/grafana/tempo/pull/6096), [6227](https://github.com/grafana/tempo/pull/6227)] - Updated `github.com/expr-lang/expr` to v1.17.7 to address [CVE-2025-68156](https://github.com/expr-lang/expr/security/advisories/GHSA-cfpf-hrx2-8rv6). \[[PRs 6230](https://github.com/grafana/tempo/pull/6230), [6092](https://github.com/grafana/tempo/pull/6092)] ### 2.8.2 - Updated Go to version 1.24.4. Addresses [CVE-2025-22874](https://nvd.nist.gov/vuln/detail/CVE-2025-22874), [CVE-2025-4673](https://nvd.nist.gov/vuln/detail/CVE-2025-4673), [CVE-2025-0913](https://nvd.nist.gov/vuln/detail/CVE-2025-0913), and [GHSA-FV92-FJC5-JJ9H](https://www.wiz.io/vulnerability-database/cve/ghsa-fv92-fjc5-jj9h). \[[PR 5323](https://github.com/grafana/tempo/pull/5323)] ### 2.8.0 - Used distroless base container images for improved security. ([PR 4556](https://github.com/grafana/tempo/pull/4556)) - Updated to go 1.24.3. ([PR 5110](https://github.com/grafana/tempo/pull/5110)) ## Bug fixes For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). ### 2.8.3 - Fixed deadlock on invalid query to `api/v2/search/tags` (SearchTagsV2). \[[PRs 5607](https://github.com/grafana/tempo/pull/5607), [6228](https://github.com/grafana/tempo/pull/6228)] - Fixed panic in `query_range` HTTP handling that could be triggered by cancellations or other errors. \[[PRs 5667](https://github.com/grafana/tempo/pull/5667), [6229](https://github.com/grafana/tempo/pull/6229)] - Fixed configuration options that were always overridden by runtime overrides, preventing users from setting certain options without using the overrides configuration. \[[PR 5202](https://github.com/grafana/tempo/pull/5202)] - Fixed trace idle period in ingesters, which was not being correctly applied. Added a max live trace period (default 30s) to prevent very large traces from being held in memory indefinitely when spans are drip-fed over time. \[[PR 5346](https://github.com/grafana/tempo/pull/5346)] - Fixed the `SearchTagValuesV2` endpoint to return 400 Bad Request instead of 500 Internal Server Error when an invalid tag name is provided. \[[PR 5493](https://github.com/grafana/tempo/pull/5493)] ### 2.8.2 - Added a nil check to `partitionAssignmentVar`. \[[PR 5198](https://github.com/grafana/tempo/pull/5198)] - Corrected instant query calculation. \[[PR 5252](https://github.com/grafana/tempo/pull/5252)] - Fixed tracing context propagation in distributor HTTP write requests. \[[PR 5312](https://github.com/grafana/tempo/pull/5312)] - Fixed search by `trace:id` with short trace ID. \[[PR 5331](https://github.com/grafana/tempo/pull/5331)] - Fixed bug where `most_recent=true` wouldn’t return most recent results when query overlapped ingesters and few other blocks. \[[PR 5438](https://github.com/grafana/tempo/pull/5438)] - Fixed a panic when the counter series was missing during `avg_over_time` aggregation. \[[PR 5300](https://github.com/grafana/tempo/pull/5300)] ### 2.8.1 - Fixed an ingester issue where a hash collision could lead to spans stored incorrectly. ([PR 5276](https://github.com/grafana/tempo/pull/5276)) ### 2.8.0 - Chose a default step for a gRPC streaming query range request if none is provided. Correctly copy exemplars for metrics like `| rate()` when gRPC streaming. ([PR 4576](https://github.com/grafana/tempo/pull/4576)) - Fixed distributor issue where a hash collision could lead to spans stored incorrectly ([PR 5186](https://github.com/grafana/tempo/pull/5186)) - Added object name to cache key in ReadRange ([PR 4982](https://github.com/grafana/tempo/pull/4982)) - Fixed rare panic during compaction ([PR 4915](https://github.com/grafana/tempo/pull/4915)) - Returned the operand as the only value if the tag is already filtered in the query ([PR 4673](https://github.com/grafana/tempo/pull/4673)) - Included cost attribution when converting from default configuration to legacy one. ([PR 4937](https://github.com/grafana/tempo/pull/4937)) - Updated `memcached` to respect cancelled context to prevent panic. ([PR 5041](https://github.com/grafana/tempo/pull/5041)) - Fixed setting processors in user configurations overrides via API. ([PR 4741](https://github.com/grafana/tempo/pull/4741)) - Fixed panic on startup. ([PR 4744](https://github.com/grafana/tempo/pull/4744)) - Fixed intrinsic tag lookups dropped when max tag lookup response size is exceeded. ([PR 4784](https://github.com/grafana/tempo/pull/4784)) - Fixed error propagation in the SyncIterator. ([PR 5045](https://github.com/grafana/tempo/pull/5045)) - Fixed mixin to include `otlp_v1_traces` HTTP write route ([PR 5072](https://github.com/grafana/tempo/pull/5072)) - Fixed `TempoBlockListRisingQuickly` alert grouping. ([PR 4876](https://github.com/grafana/tempo/pull/4876)) - Fixed metrics generator host info processor overrides configuration. ([PR 5118](https://github.com/grafana/tempo/pull/5118)) - Fixed metrics generator `target_info` to skip attributes with no name to prevent downstream errors. ([PR 5148](https://github.com/grafana/tempo/pull/5148))