Version 1.4 release notes

The Tempo team is excited to announce the release of Tempo 1.4.

Features and enhancements

Server-side metrics As the headline feature for Tempo 1.4, Server-side metrics generate metrics from your traces, for advanced insight into your application’s performance over time.
Start and end time start and end parameters were added to the /api/traces/{traceID} endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. PR 1388
Protect instances from very large traces Additional protection against large traces has been added. These features all key off the existing max_bytes_per_trace configuration parameter. PR 1317 and PR 1318
Hedge everything Improve backend search throughput Tempo using our favorite solution: hedging.
```
querier:
  search:
    external_hedge_requests_at: 5s
    external_hedge_requests_up_to: 3
```

When upgrading to Tempo v1.4, be aware of these breaking changes:

PR 1227 A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.

PR 1350 A few querier options were moved while adding hedging to external endpoints.

querier:
search_query_timeout: 30s
search_external_endpoints: []
search_prefer_self: 2

was relocated to

querier:
search:
    query_timeout: 30s
    prefer_self: 2
    external_endpoints: []

PR 1297 Updated vulture testing of full backend search, and as a result, dropped the CLI parameter tempo-search-retention-duration.

PR 1390 Correct the handling of Azure “Blob Not Found” errors.
PR 1314 Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.
PR 1387 Eliminate a data race and ingester crash during search by trace ID.
PR 1389 Eliminate spurious “failed to mark block compacted during retention” errors.
PR 1379 Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.

PR 1417 Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.
PR 1420 Compactor: populate compaction_objects_combined_total and tempo_discarded_spans_total{reason="trace_too_large_to_compact"} metrics again.
PR 1422 Distributor: prevent panics when concurrently calling shutdown to forwarder’s queueManager.