---
title: "Version 1.4 release notes | Grafana Tempo documentation"
description: "Release notes for Grafana Tempo 1.4"
---

# Version 1.4 release notes

The Tempo team is excited to announce the release of Tempo 1.4.

## Features and enhancements

- **Server-side metrics** As the headline feature for Tempo 1.4, [Server-side metrics](../../../metrics-from-traces/metrics-generator/) generate metrics from your traces, for advanced insight into your application’s performance over time.
- **Start and end time** `start` and `end` parameters were added to the `/api/traces/{traceID}` endpoint. This allows more efficient searches by only querying a subset of blocks on the backend. [PR 1388](https://github.com/grafana/tempo/pull/1388)
- **Protect instances from very large traces** Additional protection against large traces has been added. These features all key off the existing `max_bytes_per_trace` configuration parameter. [PR 1317](https://github.com/grafana/tempo/pull/1317) and [PR 1318](https://github.com/grafana/tempo/pull/1318)
- **Hedge everything** Improve backend search throughput Tempo using our favorite solution: hedging.
  
  ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```none
  querier:
    search:
      external_hedge_requests_at: 5s
      external_hedge_requests_up_to: 3
  ```

## Upgrade considerations

When upgrading to Tempo v1.4, be aware of these breaking changes:

- [PR 1227](https://github.com/grafana/tempo/pull/1227) A new ingestion endpoint was added to support faster search and proper start/end times on blocks. Roll out all ingesters before rolling the distributors to prevent downtime. Also, during this period, the ingesters will use considerably more resources and should be scaled up, or incoming traffic should be heavily throttled. Once all distributors and ingesters have rolled, performance will return to normal. Internally, we have observed ~1.5x CPU load on the ingesters during the rollout.
- [PR 1350](https://github.com/grafana/tempo/pull/1350) A few querier options were moved while adding hedging to external endpoints.
  
  YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```yaml
  querier:
  search_query_timeout: 30s
  search_external_endpoints: []
  search_prefer_self: 2
  ```
  
  was relocated to
  
  YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```yaml
  querier:
  search:
    query_timeout: 30s
    prefer_self: 2
    external_endpoints: []
  ```
- [PR 1297](https://github.com/grafana/tempo/pull/1297) Updated vulture testing of full backend search, and as a result, dropped the CLI parameter `tempo-search-retention-duration`.

## Bug fixes

### 1.4.0 bug fixes

- [PR 1390](https://github.com/grafana/tempo/pull/1390) Correct the handling of Azure “Blob Not Found” errors.
- [PR 1314](https://github.com/grafana/tempo/pull/1314) Correct block start and end times to be based on actual trace start/end times, instead of on trace ingestion times.
- [PR 1387](https://github.com/grafana/tempo/pull/1387) Eliminate a data race and ingester crash during search by trace ID.
- [PR 1389](https://github.com/grafana/tempo/pull/1389) Eliminate spurious “failed to mark block compacted during retention” errors.
- [PR 1379](https://github.com/grafana/tempo/pull/1379) Eliminate an error with the message “Writer is closed” by resetting the compression writer correctly on the error path.

### 1.4.1 bug fixes

- [PR 1417](https://github.com/grafana/tempo/pull/1417) Metrics-generator: don’t inject X-Scope-OrgID header for single-tenant setups.
- [PR 1420](https://github.com/grafana/tempo/pull/1420) Compactor: populate `compaction_objects_combined_total` and `tempo_discarded_spans_total{reason="trace_too_large_to_compact"}` metrics again.
- [PR 1422](https://github.com/grafana/tempo/pull/1422) Distributor: prevent panics when concurrently calling `shutdown` to forwarder’s queueManager.