Version 1 release notes on Grafana Labs

Version 1.5 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 1.5 release notes

The Tempo team is pleased to announce the release of Tempo 1.5.

With this release, Tempo adds experimental support for a new trace storage format based on Apache Parquet, anonymous usage reporting to help the Tempo team understand how Tempo is used, and enhancements to the metrics-generator.

We encourage users to upgrade to Grafana 9.1 to take advantage of new Tempo-powered capabilities, such as the service graph view which visualizes span metrics (traces data for rates, error rates, and durations (RED)) and service graphs.

Note: For a complete list of changes and enhancements, refer to the Tempo 1.5 changelog.

Features and enhancements

Version 1.5 contains the following features and enhancements.

Apache Parquet block format support

Tempo now supports a columnar block format for trace storage based on Apache Parquet. We’re excited about this new format because it will enable increased speed and efficiency for Tempo search. It also allows users to leverage the large ecosystem of existing Parquet tools and libraries to process and transform their trace data.

We’re releasing the Parquet block format as experimental in Tempo 1.5, with plans to move it to stable in a future release once we’ve had more opportunity to test it.

The new Parquet block format can be used as a drop-in replacement for Tempo’s existing block format. No data conversion or upgrade process is necessary. As soon as the Parquet format is enabled, Tempo starts writing data in that format, leaving existing data as-is. For more information, refer to the Parquet documentation. Please note, however, that enabling the Parquet block format means Tempo will require more CPU and memory resources than it previously did.

To use Parquet, set the block format option to vParquet in the configuration file. Refer to the Parquet documentation and the Configuration documentation.

# block format version. options: v2, vParquet
[version: <string> | default = v2]

For more information about the Parquet implementation, refer to the Parquet design document and Issue 1480.

Metrics-generator enhancements

The metrics-generator now includes databases and requests across messaging systems in service graph metrics. The OpenTelemetry semantic conventions are used to detect when a span is involved in one of these systems. PR 1576

In addition, metrics-generator supports per-tenant processor configuration. PR 1434

Anonymous usage reporting

By default, Tempo reports anonymous usage data about itself to Grafana Labs. This data is used to understand which features are commonly enabled, as well as which deployment modes, replication factors, and compression levels are most popular, etc.

By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts. No private information is collected, and all reports are completely anonymous.

Reporting is controlled by the usage_report configuration option and can be disabled. For instructions, refer to the Configuration documentation. PR 1465

Upgrade considerations

When upgrading to Tempo v1.5, be aware of these breaking changes:

PR 1478 To build advanced visualization features into Grafana, we have decided to change our spanmetric names to match OpenTelemetry (OTel) conventions. Any functionality added to Grafana will work whether you use Tempo, Grafana Agent, or the OTel Collector to generate metrics. The old histogram metric name in Tempo 1.4.x:
```
traces_spanmetrics_duration_seconds_{sum,count,bucket}
```
has been replaced with this new name in Tempo 1.5:
```
traces_spanmetrics_latency_{sum,count,bucket}
```
Additionally, the default label span_status has been renamed to status_code.

PR 1556 Jsonnet users will need to specify ephemeral storage requests and limits for the metrics generator.

 _config+:: {
   metrics_generator+: {
     ephemeral_storage_request_size: '10Gi',
     ephemeral_storage_limit_size: '11Gi',
   },
 }

PR 1481 As a result of the anonymous usage reporting feature, distributors and metrics-generators now require the same level of access to object storage as compactors and ingesters. Users who disable anonymous usage reporting are exempt from this requirement.
PR 1558 Deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds have been removed in favor of tempodb_backend_request_duration_seconds. These metrics have been deprecated since v1.1.

Bug fixes

1.5.0 bug fixes

Version 1.5.0 includes the following fixes:

PR 1441 Fixed nil pointer panic when the trace by id path errors.
PR 1472 Updated Tempo microservices Helm values example which missed the enabled key for thriftHttp.
PR 1468 Fixed race condition in forwarder overrides loop.
PR 1538 Fixed v2 backend check on span name to be substring.
PR 1548 Fixed wal check on span name to be substring.
Issue 1258 Prevented the ingester panic “cannot grow buffer”.
PR 1554 metrics-generator: do not remove x-scope-orgid header in single tenant modus.
PR 1589 Fixed an issue where backend does not support root.name and root.service.name.
Issue 1603 Fixed the ingester to continue starting up after block replay error.

Version 1.4 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 1.4 release notes

The Tempo team is excited to announce the release of Tempo 1.4.

Features and enhancements

Server-side metrics As the headline feature for Tempo 1.4, Server-side metrics generate metrics from your traces, for advanced insight into your application’s performance over time.
Start and end time start and end parameters were added to the /api/traces/{traceID} endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. PR 1388
Protect instances from very large traces Additional protection against large traces has been added. These features all key off the existing max_bytes_per_trace configuration parameter. PR 1317 and PR 1318
Hedge everything Improve backend search throughput Tempo using our favorite solution: hedging.
```
querier:
  search:
    external_hedge_requests_at: 5s
    external_hedge_requests_up_to: 3
```

Upgrade considerations

When upgrading to Tempo v1.4, be aware of these breaking changes:

PR 1227 A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.

PR 1350 A few querier options were moved while adding hedging to external endpoints.

querier:
search_query_timeout: 30s
search_external_endpoints: []
search_prefer_self: 2

was relocated to

querier:
search:
  query_timeout: 30s
  prefer_self: 2
  external_endpoints: []

PR 1297 Updated vulture testing of full backend search, and as a result, dropped the CLI parameter tempo-search-retention-duration.

Bug fixes

1.4.0 bug fixes

PR 1390 Correct the handling of Azure “Blob Not Found” errors.
PR 1314 Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.
PR 1387 Eliminate a data race and ingester crash during search by trace ID.
PR 1389 Eliminate spurious “failed to mark block compacted during retention” errors.
PR 1379 Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.

1.4.1 bug fixes

PR 1417 Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.
PR 1420 Compactor: populate compaction_objects_combined_total and tempo_discarded_spans_total{reason="trace_too_large_to_compact"} metrics again.
PR 1422 Distributor: prevent panics when concurrently calling shutdown to forwarder’s queueManager.

Version 1.3 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 1.3 release notes

The Tempo team is excited to announce the release of Tempo 1.3.

Features and enhancements

Search In addition to the existing search of recent traces, support has been added to search traces in the backend store. Refer to Tempo search for details.
Improve ring functionality The compactor ring automatically forgets about unhealthy ring members.
Protect instances from very large traces The ingester keeps track of traces that exceed the configured maximum size, extending protection from very large traces to after the traces are flushed to disk.
Expose additional metrics These new metrics are exposed:
- tempo_limits_overrides
- tempo_limits_defaults
- tempo_ingester_live_traces
- tempodb_compaction_outstanding_blocks
Azure credentials may be set in the environment Instead of setting values in the configuration, environment variables may specify AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY.

Upgrade considerations

When upgrading to Tempo v1.3, be aware of these breaking changes:

PR 1142: The OpenTelemetry Collector’s GRPC receiver’s default port changed from 55680 to 4317.
PR 1174: Configuration option querier.search_max_result_limit is renamed to query_frontend.search.max_result_limit. Configuration option querier.search_default_result_limit is renamed to query_frontend.search.default_result_limit.
PR 1173: Removed the deprecated Push method from tempopb.Pusher.

Known issues

Grafana does not yet support the tempoBackendSearch feature toggle to enable searches in the backend datastore.

Bug fixes

1.3.1 bug fixes

PR 1260 mapno: Fixed a panic that occurred when using etcd as the ring’s backend storage.

1.3.2 bug fixes

PR 1295 joe-elliott: Fixed an issue which corrupted the start/end time ranges of Tempo searches that targeted ingesters.

Version 1.2 release notes

Thu, 09 Apr 2026 14:59:14 +0000

Version 1.2 release notes

The Tempo team is excited to announce the release of Tempo 1.2. This release introduces Tempo search.

Features and enhancements

Recent traces search The headline feature of Tempo 1.2 is the ability to search for data that is still in the ingester component. This experimental feature must be enabled using feature flags in both Grafana and Tempo. Refer to Tempo search for details.
Scalable monolithic deployment mode (formerly scalable single binary deployment mode) The new scalable monolithic deployment mode is operationally simpler, although less robust than a fully distributed deployment. We consider it a balanced operational mode that some may find attractive.
Massive performance improvements were achieved by more efficiently batching trace data being ingested. In terms of CPU needed, Tempo v1.2 is twice as efficient as Tempo v1.1. Refer to PR 1075 for details.
API improvements Some informational endpoints were consolidated within the status endpoint. New endpoints were also added.

Upgrade considerations

When upgrading to Tempo v1.2, be aware of these changes:

Support for v0 and v1 blocks were dropped as announced with Tempo v1.1. Refer to the Tempo v1.1 changelog for details.
PR 1007 altered the Querier API. When upgrading components within a deployment, there will be a read outage until all queriers and query frontends have rolled to the upgraded version.
API improvements consolidated informational endpoints within the status endpoint.
Metric ingester_bytes_metric_total is renamed ingester_bytes_received_total by PR 979.
Metric cortex_runtime_config_last_reload_successful is renamed tempo_runtime_config_last_reload_successful by PR 945.
The tempo-cli flag --storage.trace.maintenance-cycle is renamed -storage.trace.blocklist_poll by PR 897.

Bug fixes

1.2.1 bug fixes

Honor the default values of configuration parameters max_bytes_per_trace and max_search_bytes_per_trace. PR #1109 (@BitProcessor)
Reclassify the writing of empty traces to be a warning instead of an error. PR #1113 (@mdisibio)

1.2.0 bug fixes

Fixed errors with magic numbers and other block mishandling when an ingester forcefully shuts down. Issue #937 (@mdisibio)
Fixed a memory leak in the compactor component. PR #806 (@mdisibio)
Set a span’s tag span.kind to client in query-frontend to allow query frontend spans to be paired with querier server spans. PR #975 (@mapno)
Metric tempodb_backend_hedged_roundtrips_total now correctly counts hedged roundtrips. PR #1079 (@joe-elliott)
Updated the go-kit logger package to remove spurious debug logs. PR #1094 (@bboreham)