This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.
Version 1.4 release notes
The Tempo team is excited to announce the release of Tempo 1.4.
Features and enhancements
Server-side metrics As the headline feature for Tempo 1.4, Server-side metrics generate metrics from your traces, for advanced insight into your application’s performance over time.
Start and end time
start
andend
parameters were added to the/api/traces/{traceID}
endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. PR 1388Protect instances from very large traces Additional protection against large traces has been added. These features all key off the existing
max_bytes_per_trace
configuration parameter. PR 1317 and PR 1318Hedge everything Improve backend search throughput Tempo using our favorite solution: hedging.
querier: search: external_hedge_requests_at: 5s external_hedge_requests_up_to: 3
Upgrade considerations
When upgrading to Tempo v1.4, be aware of these breaking changes:
PR 1227 A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.
PR 1350 A few querier options were moved while adding hedging to external endpoints.
querier: search_query_timeout: 30s search_external_endpoints: [] search_prefer_self: 2
was relocated to
querier: search: query_timeout: 30s prefer_self: 2 external_endpoints: []
PR 1297 Updated vulture testing of full backend search, and as a result, dropped the CLI parameter
tempo-search-retention-duration
.
Bug fixes
1.4.0 bug fixes
PR 1390 Correct the handling of Azure “Blob Not Found” errors.
PR 1314 Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.
PR 1387 Eliminate a data race and ingester crash during search by trace ID.
PR 1389 Eliminate spurious “failed to mark block compacted during retention” errors.
PR 1379 Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.
1.4.1 bug fixes
PR 1417 Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.
PR 1420 Compactor: populate
compaction_objects_combined_total
andtempo_discarded_spans_total{reason="trace_too_large_to_compact"}
metrics again.PR 1422 Distributor: prevent panics when concurrently calling
shutdown
to forwarder’s queueManager.