Version 1.5 release notes

The Tempo team is pleased to announce the release of Tempo 1.5.

With this release, Tempo adds experimental support for a new trace storage format based on Apache Parquet, anonymous usage reporting to help the Tempo team understand how Tempo is used, and enhancements to the metrics-generator.

We encourage users to upgrade to Grafana 9.1 to take advantage of new Tempo-powered capabilities, such as the service graph view which visualizes span metrics (traces data for rates, error rates, and durations (RED)) and service graphs.

Note: For a complete list of changes and enhancements, refer to the Tempo 1.5 changelog.

Features and enhancements

Version 1.5 contains the following features and enhancements.

Apache Parquet block format support

Tempo now supports a columnar block format for trace storage based on Apache Parquet. We’re excited about this new format because it will enable increased speed and efficiency for Tempo search. It also allows users to leverage the large ecosystem of existing Parquet tools and libraries to process and transform their trace data.

We’re releasing the Parquet block format as experimental in Tempo 1.5, with plans to move it to stable in a future release once we’ve had more opportunity to test it.

The new Parquet block format can be used as a drop-in replacement for Tempo’s existing block format. No data conversion or upgrade process is necessary. As soon as the Parquet format is enabled, Tempo starts writing data in that format, leaving existing data as-is. For more information, refer to the Parquet documentation. Please note, however, that enabling the Parquet block format means Tempo will require more CPU and memory resources than it previously did.

To use Parquet, set the block format option to vParquet in the configuration file. Refer to the Parquet documentation and the Configuration documentation.

# block format version. options: v2, vParquet
[version: <string> | default = v2]

For more information about the Parquet implementation, refer to the Parquet design document and Issue 1480.

Metrics-generator enhancements

The metrics-generator now includes databases and requests across messaging systems in service graph metrics. The OpenTelemetry semantic conventions are used to detect when a span is involved in one of these systems. PR 1576

In addition, metrics-generator supports per-tenant processor configuration. PR 1434

Anonymous usage reporting

By default, Tempo will report anonymous usage data about itself to Grafana Labs. This data is used to understand which features are commonly enabled, as well as which deployment modes, replication factors, and compression levels are most popular, etc.

By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts. No private information is collected, and all reports are completely anonymous.

Reporting is controlled by the usage_report configuration option and can be disabled. For instructions, refer to the Configuration documentation. PR 1465

Upgrade considerations

When upgrading to Tempo v1.5, be aware of these breaking changes:

PR 1478 To build advanced visualization features into Grafana, we have decided to change our spanmetric names to match OpenTelemetry (OTel) conventions. Any functionality added to Grafana will work whether you use Tempo, Grafana Agent, or the OTel Collector to generate metrics. The old histogram metric name in Tempo 1.4.x:
```
traces_spanmetrics_duration_seconds_{sum,count,bucket}
```
has been replaced with this new name in Tempo 1.5:
```
traces_spanmetrics_latency_{sum,count,bucket}
```
Additionally, the default label span_status has been renamed to status_code.

PR 1556 Jsonnet users will need to specify ephemeral storage requests and limits for the metrics generator.

 _config+:: {
   metrics_generator+: {
     ephemeral_storage_request_size: '10Gi',
     ephemeral_storage_limit_size: '11Gi',
   },
 }

PR 1481 As a result of the anonymous usage reporting feature, distributors and metrics-generators now require the same level of access to object storage as compactors and ingesters. Users who disable anonymous usage reporting are exempt from this requirement.
PR 1558 Deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds have been removed in favor of tempodb_backend_request_duration_seconds. These metrics have been deprecated since v1.1.

Bug fixes

1.5.0 bug fixes

Version 1.5.0 includes the following fixes:

PR 1441 Fixed nil pointer panic when the trace by id path errors.
PR 1472 Updated Tempo microservices Helm values example which missed the enabled key for thriftHttp.
PR 1468 Fixed race condition in forwarder overrides loop.
PR 1538 Fixed v2 backend check on span name to be substring.
PR 1548 Fixed wal check on span name to be substring.
Issue 1258 Prevented the ingester panic “cannot grow buffer”.
PR 1554 metrics-generator: do not remove x-scope-orgid header in single tenant modus.
PR 1589 Fixed an issue where backend does not support root.name and root.service.name.
Issue 1603 Fixed the ingester to continue starting up after block replay error.