Version 1.4 release notes

The Tempo team is excited to announce the release of Tempo 1.4.

Features and enhancements

• Server side metrics As the headline feature for Tempo 1.4, Server-side metrics generate metrics from your traces, for advanced insight into your application’s performance over time.

• Start and end time start and end parameters were added to the /api/traces/{traceID} endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. PR 1388

• Protect instances from very large traces Additional protection against large traces has been added. These features all key off the existing max_bytes_per_trace configuration parameter. PR 1317 and PR 1318

• Hedge everything Improve backend search throughput Tempo using our favorite solution: hedging.

  querier:
    search:
      external_hedge_requests_at: 5s
      external_hedge_requests_up_to: 3
  

Upgrade considerations

When upgrading to Tempo v1.4, be aware of these breaking changes:

• PR 1227 A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.

• PR 1350 A few querier options were moved while adding hedging to external endpoints.

  querier:
  search_query_timeout: 30s
  search_external_endpoints: []
  search_prefer_self: 2
  

  was relocated to

  querier:
  search:
      query_timeout: 30s
      prefer_self: 2
      external_endpoints: []
  

• PR 1297 Updated vulture testing of full backend search, and as a result, dropped the CLI parameter tempo-search-retention-duration.

Bug fixes

1.4.0 bug fixes

• PR 1390 Correct the handling of Azure “Blob Not Found” errors.

• PR 1314 Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.

• PR 1387 Eliminate a data race and ingester crash during search by trace ID.

• PR 1389 Eliminate spurious “failed to mark block compacted during retention” errors.

• PR 1379 Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.

1.4.1 bug fixes

• PR 1417 Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.

• PR 1420 Compactor: populate compaction_objects_combined_total and tempo_discarded_spans_total{reason="trace_too_large_to_compact"} metrics again.

• PR 1422 Distributor: prevent panics when concurrently calling shutdown to forwarder’s queueManager.