Version 1.4 release notes
The Tempo team is excited to announce the release of Tempo 1.4.
Features and enhancements
-
Server side metrics As the headline feature for Tempo 1.4, Server-side metrics generate metrics from your traces, for advanced insight into your application’s performance over time.
-
Start and end time
start
andend
parameters were added to the/api/traces/{traceID}
endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. PR 1388 -
Protect instances from very large traces Additional protection against large traces has been added. These features all key off the existing
max_bytes_per_trace
configuration parameter. PR 1317 and PR 1318 -
Hedge everything Improve backend search throughput Tempo using our favorite solution: hedging.
querier: search: external_hedge_requests_at: 5s external_hedge_requests_up_to: 3
Upgrade considerations
When upgrading to Tempo v1.4, be aware of these breaking changes:
-
PR 1227 A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.
-
PR 1350 A few querier options were moved while adding hedging to external endpoints.
querier: search_query_timeout: 30s search_external_endpoints: [] search_prefer_self: 2
was relocated to
querier: search: query_timeout: 30s prefer_self: 2 external_endpoints: []
-
PR 1297 Updated vulture testing of full backend search, and as a result, dropped the CLI parameter
tempo-search-retention-duration
.
Bug fixes
1.4.0 bug fixes
-
PR 1390 Correct the handling of Azure “Blob Not Found” errors.
-
PR 1314 Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.
-
PR 1387 Eliminate a data race and ingester crash during search by trace ID.
-
PR 1389 Eliminate spurious “failed to mark block compacted during retention” errors.
-
PR 1379 Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.
1.4.1 bug fixes
-
PR 1417 Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.
-
PR 1420 Compactor: populate
compaction_objects_combined_total
andtempo_discarded_spans_total{reason="trace_too_large_to_compact"}
metrics again. -
PR 1422 Distributor: prevent panics when concurrently calling
shutdown
to forwarder’s queueManager.
Related Tempo resources
GrafanaCONline 2021
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.
Tracing made simple with Grafana
Dive into the new options for viewing tracing data inside Grafana and learn how to make tracing an integral part of your observability strategy.