This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.
Configure TraceQL metrics
TraceQL language provides metrics queries as a feature. Metric queries extend trace queries by applying a function to trace query results. This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.
Before you begin
To use the metrics generated from traces, you need to:
- Configure a Tempo data source in Grafana or Grafana Cloud ( documentation)
- Access Grafana Cloud or Grafana version 10.4 or later
Evaluate query timeouts
Because of their expensive nature, these queries can take a long time to run. As such, consider increasing the timeouts in various places of the system to allow enough time for the data to be returned.
Consider these areas when raising timeouts:
- Any proxy in front of Grafana
- Grafana data source for Prometheus pointing at Tempo
- Tempo configuration
querier.search.query_timeoutserver.http_server_read_timeoutserver.http_server_write_timeout
Set TraceQL metrics query options
The query_frontend.metrics configuration block controls all TraceQL metrics queries.
The configuration depends on the environment.
Note
The default maximum time range for a metrics query is 24 hours, configured using the
query_frontend.metrics.max_durationparameter.This is different to the default TraceQL maximum time range of 168 hours (7 days).
max_durationgives every tenant the same range, but data volume differs by orders of magnitude between tenants.Use
query_frontend.metrics.max_scanned_bytesto limit a query by the bytes of the backend blocks it scans. It allows tenants to query time ranges betweenmin_durationandmax_durationthat scan up tomax_scanned_bytes.A query of
min_durationor less is always allowed, and a longer query over the allowed range is rejected with a 400 error.min_durationmust not be abovemax_duration. Only backend blocks count towardmax_scanned_bytes, so data newer thanquery_backend_afterisn’t counted. The limit counts the requested range, so step alignment can add up to 4 steps of data on top ofmax_scanned_bytes. The default of0disables the limit. Search has the same settings underquery_frontend.search. Refer to Query range limits for details.
The query_frontend.metrics.query_backend_after parameter controls the boundary between querying the live-store and backend storage.
Time ranges older than query_backend_after (default 15m) are searched in backend/object storage only, while more recent data is queried from the live-store.
For example, in a cloud environment, smaller jobs with more concurrency may be desired due to the nature of scale on the backend.
query_frontend:
metrics:
concurrent_jobs: 1000
target_bytes_per_job: 2.25e+08 # ~225MB
interval: 30m0sFor an on-prem backend, you can improve query times by lowering the concurrency, while increasing the job size.
query_frontend:
metrics:
concurrent_jobs: 8
target_bytes_per_job: 1.25e+09 # ~1.25GBSampling and performance optimization
TraceQL metrics queries support sampling hints to improve performance on large datasets. Refer to the TraceQL metrics sampling documentation for more information.
When using sampling in your TraceQL metrics queries, consider:
- Timeout settings: Sampled queries run faster but may still benefit from adequate timeouts
- Concurrent jobs: Sampling reduces per-job processing time, allowing higher concurrency


