TraceQL metrics on Grafana Labs

Configure TraceQL metrics

Thu, 09 Apr 2026 14:59:14 +0000

Configure TraceQL metrics

Caution
TraceQL metrics is an public preview feature. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available TraceQL metrics are enabled by default in Grafana Cloud.

TraceQL language provides metrics queries as an experimental feature. Metric queries extend trace queries by applying a function to trace query results. This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs.

Before you begin

To use the metrics generated from traces, you need to:

Set the local-blocks processor to active in your metrics-generator configuration
Configure a Tempo data source in Grafana or Grafana Cloud ( documentation)
Access Grafana Cloud or Grafana version 10.4 or later

Refer to the Metrics-generator configuration documentation for more information about the metrics-generator configuration.

Activate and configure the `local-blocks` processor

You must enable the local-blocks processor to start using metrics queries like { } | rate(). If not enabled, then the metrics queries fail with the error localblocks processor not found. Enabling the local-blocks processor can be done either per tenant or in all tenants.

To activate the local-blocks processor for all users, add it to the list of processors in the overrides block of your Tempo configuration.

# Global overrides configuration.
overrides:
  metrics_generator_processors: ["local-blocks"]

To configure the processor per tenant, use the metrics_generator_processor override.

Example for per-tenant in the per-tenant overrides:

overrides:
  'tenantID':
    metrics_generator_processors:
      - local-blocks

By default, for all tenants in the main configuration:

overrides:
  defaults:
    metrics_generator:
      processors: [local-blocks]

Add this configuration to run TraceQL metrics queries against all spans and not just server spans:

metrics_generator:
  processor:
    local_blocks:
      filter_server_spans: false

To run metrics queries on historical data, you must configure the local-blocks processor to flush RF1 blocks to object storage:

metrics_generator:
  processor:
    local_blocks:
      flush_to_storage: true

Setting flush_to_storage to true ensures that metrics blocks are flushed to storage so TraceQL metrics queries against historical data.

If you configured Tempo using the tempo-distributed Helm chart, you can also set traces_storage using your values.yaml file. Refer to the Helm chart for an example.

For more information about overrides, refer to Standard overrides.

Local blocks and metrics-generator in Azure blob storage and Helm

Note
This configuration only applies if you are using a Helm chart, like tempo-distributed, to deploy Tempo.

By default, the metrics-generator doesn’t require a backend connection unless you’ve enabled the local_blocks processor. The local_blocks processor is used for generating metrics from traces, which is required for TraceQL metrics When this configuration is set, the metrics-generator produces blocks and flushes them into a backend storage.

In this case, list the generator in the env var expansion configuration so the STORAGE_ACCOUNT_ACCESS_KEY has the secret value.

You can use this configuration example with Helm charts, like tempo-distributed. Replace any values in all caps with the values for your Helm deployment.

generator:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: <STORAGE_ACCOUNT_ACCESS_KEY>
      valueFrom:
        secretKeyRef:
          name: <TEMPO-TRACES-STG-KEY>
          key: <TEMPO-TRACES-KEY>

For more information, refer to Azure hosted storage.

Evaluate query timeouts

Because of their expensive nature, these queries can take a long time to run. As such, consider increasing the timeouts in various places of the system to allow enough time for the data to be returned.

Consider these areas when raising timeouts:

Any proxy in front of Grafana
Grafana data source for Prometheus pointing at Tempo
Tempo configuration
- querier.search.query_timeout
- server.http_server_read_timeout
- server.http_server_write_timeout

Set TraceQL metrics query options

The query_frontend.metrics configuration block controls all TraceQL metrics queries. The configuration depends on the environment.

Note
The default maximum time range for a metrics query is 3 hours, configured using the query_frontend.metrics.max_duration parameter.

This is different to the default TraceQL maximum time range of 168 hours (7 days).

For example, in a cloud environment, smaller jobs with more concurrency may be desired due to the nature of scale on the backend.

query_frontend:
  metrics:
    concurrent_jobs: 1000
    target_bytes_per_job: 2.25e+08 # ~225MB
    interval: 30m0s

For an on-prem backend, you can improve query times by lowering the concurrency, while increasing the job size.

query_frontend:
  metrics:
    concurrent_jobs: 8
    target_bytes_per_job: 1.25e+09 # ~1.25GB

Sampling and performance optimization

TraceQL metrics queries support sampling hints to improve performance on large datasets. Refer to the TraceQL metrics sampling documentation for more information.

When using sampling in your TraceQL metrics queries, consider:

Timeout settings: Sampled queries run faster but may still benefit from adequate timeouts
Concurrent jobs: Sampling reduces per-job processing time, allowing higher concurrency

TraceQL metrics sampling

Thu, 09 Apr 2026 14:59:14 +0000

TraceQL metrics sampling

Caution
TraceQL metrics is an public preview feature. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available TraceQL metrics are enabled by default in Grafana Cloud.

TraceQL metrics sampling dynamically and automatically chooses how to sample your tracing data to give you the highest quality signal with examining as little data as possible. The overall performance improvement depends on the query. Heavy queries, such as { } | rate(), show improvements of 2-4 times.

Sampling intelligently selects a representative subset of data for processing, making it particularly valuable for:

Real-time dashboards requiring fast refresh rates
Exploratory data analysis where approximate results accelerate insights
Resource-constrained environments with limited compute capacity
Large-scale deployments processing terabytes of trace data daily

Adaptive sampling was featured in the September 2025 Tempo community call. Watch the recording starting at the 12:00 minute mark to learn more.

Refer to the TraceQL metrics documentation to learn more.

Sampling methods

There are three sampling methods available:

Dynamic sampling using with(sample=true), which automatically determines the optimal sampling strategy based on query characteristics.
Fixed span sampling using with(span_sample=0.xx), which selects the specified percentage of spans.
Fixed trace sampling using with(trace_sample=0.xx), which selects complete traces for analysis.
Fixed probabilistic sampling using with(sample=0.xx).

How dynamic sampling works

Dynamic sampling, with(sample=true), applies probabilistic sampling at the storage layer. This sampling method uses an adaptive probabilistic approach that responds to how common spans and traces matching the query are. This approach applies probabilistic sampling at the storage layer, for example, only inspecting xx% spans, or xx% traces, depending on the needs of the query.

When there is a lot of data, it lowers the sampling rate. When matches are rare it keeps the sampling rate higher, possibly never going below 100%. Therefore, the performance gain depends on the query.

This behavior can be overridden to focus more on fixed span sampling using with(span_sample=0.xx) or fixed trace sampling using with(trace_sample=0.xx).

Before you begin

TraceQL metrics sampling requires:

Tempo 2.8+ with TraceQL metrics enabled
local-blocks processor configured in metrics-generator ( documentation)
Grafana 10.4+ or Grafana Cloud for UI integration

You can use the TraceQL query editor in the Tempo data source in Grafana or Grafana Cloud to run the sample queries. Refer to TraceQL queries in Grafana for more information.

Dynamic sampling using `with(sample=true)`

Dynamic sampling automatically determines the optimal sampling strategy based on query characteristics. It switches between span-level and trace-level sampling as needed and adjusts sampling rates dynamically. The goal is for with(sample=true) to be safe to include in virtually any query, regardless of scale or selectivity.

{ resource.service.name="checkout-service" } | rate() with(sample=true)
{ status=error } | count_over_time() by (resource.service.name) with(sample=true)

Best for: Most queries. Specifically, all queries returning a single series, and cases where the dynamic sampling rate is important, such as when the traffic has large variations across time or is not known in advance.

Limitations: May under-sample rare events depending on the query, if it returns time series with a large difference between the most common and rarest events.

Fixed span sampling using `with(span_sample=0.xx)`

Fixed span sampling selects the specified percentage of spans.

{ status=error } | rate() by (resource.service.name) with(span_sample=0.1)

Best for: Exact control over accuracy and speed when the data characteristics are known in advance.

Limitations: May miss important events during low-volume periods and not optimal for naturally selective queries.

Fixed trace sampling using `with(trace_sample=0.xx)`

Fixed trace sampling selects complete traces for analysis, preserving trace context and relationships between spans within the same request flow.

{ } >> { status=error }  | rate() by (resource.service.name) with(trace_sample=0.1)

Best for: Trace-level aggregations, service dependency mapping, and error correlation analysis.

Limitations: Not as accurate as span-level sampling when trace sizes vary significantly. Only use for queries requiring it, such as structural or spanset correlation, and prefer adaptive or span-level sampling for all others.

TraceQL metrics functions

Thu, 09 Apr 2026 14:59:14 +0000

TraceQL metrics functions

Caution
TraceQL metrics is an public preview feature. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available TraceQL metrics are enabled by default in Grafana Cloud.

TraceQL metrics query functions are aggregate operators that can be appended to any TraceQL span selector to compute time-series metrics directly from trace data. You can answer questions about system behavior by aggregating trace data on-the-fly.

Available functions

TraceQL supports rate, count_over_time, sum_over_time, min_over_time, avg_over_time, quantile_over_time, histogram_over_time, and compare functions. These methods can be appended to any TraceQL query to calculate and return the desired metrics like:

{} | rate()

Note that topk and bottomk are also supported to only return a subset of series. These can only be added after a metrics query like:

{} | rate() by (resource.service.name) | topk(10)

These functions can be added as an operator at the end of any TraceQL query.

Function	Description	Example
`rate()`	Calculates the number of matching spans per second.	`{ span:status = error } \| rate()`
`count_over_time()`	Counts the number of matching spans per time interval.	`{ span:name = "GET /:endpoint" } \| count_over_time()`
`sum_over_time()`	Sums the value for the specified attribute across all matching spans per time interval.	`{ } \| sum_over_time(span.http.response.size)`
`min_over_time()`	Returns the minimum value for the specified attribute across all matching spans per time interval.	`{ span:name = "GET /:endpoint" } \| min_over_time(span:duration)`
`max_over_time()`	Returns the maximum value for the specified attribute across all matching spans per time interval.	`{ span:name = "GET /:endpoint" } \| max_over_time(span:duration)`
`avg_over_time()`	Returns the average value for the specified attribute across all matching spans per time interval.	`{ span:name = "GET /:endpoint" } \| avg_over_time(span:duration)`
`quantile_over_time()`	The quantile of the values in the specified interval.	`{ span:name = "GET /:endpoint" } \| quantile_over_time(span:duration, .99)`
`histogram_over_time()`	Evaluate frequency distribution over time.	`{ } \| histogram_over_time(span:duration) by (span.http.target)`
`topk()`	Returns only the top `k` results from a metrics query.	`{ resource.service.name = "foo" } \| rate() by (span.http.url) \| topk(10)`
`bottomk()`	Returns only the bottom `k` results from a metrics query.	`{ resource.service.name = "foo" } \| rate() by (span.http.url) \| bottomk(10)`

Group results with `by()`

You can use the optional by() clause with metrics functions in TraceQL metrics queries to group aggregated results by specified span attributes like service name, endpoint, or status.

Without by(), you get a single aggregated value across all matching spans.

{ span:status = error } | rate()

With by(), you get separate time series for each unique combination of the grouped attributes.

{ span:status = error } | rate() by(resource.service.name)

The `rate` function

The rate function calculates the number of matching spans per second that match the given span selectors.

The following query shows the rate of errors by service and span name. This is a TraceQL specific way of gathering rate metrics that would otherwise be generated by the span metrics processor.

For example, this query:

{ span:status = error } | rate() by (resource.service.name, span:name)

Is an equivalent to using span-generated metrics and running the query.

This example calculates the rate of the erroring spans coming from the service foo. Rate is a spans/sec quantity.

{ resource.service.name = "foo" && span:status = error } | rate()

Combined with the by() operator, this can be even more powerful.

{ resource.service.name = "foo" && span:status = error } | rate() by (span.http.route)

This example still rates the erroring spans in the service foo but the metrics are broken down by HTTP route. This might let you determine that /api/sad had a higher rate of erroring spans than /api/happy, for example.

The `count_over_time` function

The count_over_time() function counts the number of matching spans per time interval. The time interval that the count is computed over is set by the step parameter.

The `step` parameter

The step parameter defines the granularity of the returned time-series. For example, step=15s returns a data point every 15s within the time range. By default, step automatically chooses a dynamic value based on the query start time and end time.

Any value used for step needs to include a duration value, such as 30s for seconds or 1m for minutes.

You can configure this parameter using Grafana Explore or using the Tempo API.

To check or change the step value using Grafana Explore:

Select your Tempo data source.
Select either the Search or TraceQL query type tab.
Expand the Metrics options to view the Step value.

This example counts the number of spans with name "GET /:endpoint" broken down by status code. You might see that there are 10 "GET /:endpoint" spans with status code 200 and 15 "GET /:endpoint" spans with status code 400.

{ span:name = "GET /:endpoint" } | count_over_time() by (span.http.status_code)

Over-time functions

The over-time functions let you aggregate numerical values across matching spans per time interval. The time interval that these functions compute over is set by the step parameter.

For more information, refer to the step API parameter.

These functions accept the numerical field that you want to aggregate on.

The `sum_over_time` function

The sum_over_time() function lets you aggregate numerical values by computing the sum value of them. The time interval that the sum is computed over is set by the step parameter.

The `min_over_time` function

The min_over_time() function lets you aggregate numerical attributes by calculating their minimum value. For example, you could choose to calculate the minimum duration of a group of spans, or you could choose to calculate the minimum value of a custom attribute you’ve attached to your spans, like span.shopping.cart.entries. The time interval that the minimum is computed over is set by the step parameter.

This example computes the minimum duration for each http.target of all spans named "GET /:endpoint". Any numerical attribute on the span is fair game.

{ span:name = "GET /:endpoint" } | min_over_time(span:duration) by (span.http.target)

This example computes the minimum status code value of all spans named "GET /:endpoint".

{ span:name = "GET /:endpoint" } | min_over_time(span.http.status_code)

The `max_over_time` function

The max_over_time() function lets you aggregate numerical values by computing the maximum value of them, such as the all important span duration. The time interval that the maximum is computed over is set by the step parameter.

This example computes the maximum duration for each http.target of all spans named "GET /:endpoint".

{ span:name = "GET /:endpoint" } | max_over_time(span:duration) by (span.http.target)

{ span:name = "GET /:endpoint" } | max_over_time(span.http.response.size)

The `avg_over_time` function

The avg_over_time() function lets you aggregate numerical values by computing the average value of them, such as the all important span duration. The time interval that the average is computed over is set by the step parameter.

This example computes the average duration for each http.status_code of all spans named "GET /:endpoint".

{ span:name = "GET /:endpoint" } | avg_over_time(span:duration) by (span.http.status_code)

{ span:name = "GET /:endpoint" } | avg_over_time(span.http.response.size)

The `quantile_over_time` function

The quantile_over_time() function lets you aggregate numerical values, such as the all important span duration, by computing quantiles over time. You can specify multiple quantiles in the same query.

The example below computes the 99th, 90th, and 50th percentile of the duration attribute on all spans with name GET /:endpoint.

{ span:name = "GET /:endpoint" } | quantile_over_time(span:duration, .99, .9, .5)

You can group by any span or resource attribute.

{ span:name = "GET /:endpoint" } | quantile_over_time(span:duration, .99) by (span.http.target)

Quantiles aren’t limited to span duration. Any numerical attribute on the span is fair game. To demonstrate this flexibility, consider this nonsensical quantile on span.http.status_code:

{ span:name = "GET /:endpoint" } | quantile_over_time(span.http.status_code, .99, .9, .5)

This computes the 99th, 90th, and 50th percentile of the values of the status_code attribute for all spans named GET /:endpoint. This is unlikely to tell you anything useful (what does a median status code of 347 mean?), but it works.

As a further example, imagine a custom attribute like span.temperature. You could use a similar query to know what the 50th percentile and 95th percentile temperatures were across all your spans.

The `histogram_over_time` function

The histogram_over_time() function lets you evaluate frequency distribution over time for numerical values, such as span duration. This function groups values into buckets and shows how frequently values fall into each bucket over time.

This example evaluates the frequency distribution of span duration, grouped by HTTP target:

{ } | histogram_over_time(span:duration) by (span.http.target)

Multi-stage metrics queries

Multi-stage metrics queries are queries that turn your spans into metrics and then perform additional operations on those metrics.

The `topk` function

The topk function lets you aggregate and process TraceQL metrics by returning only the top k results. This function is similar to its equivalent PromQL function.

When a query response is larger than the maximum, you can use topk to return only the specified number from 1 through k of the top results.

For example:

{ resource.service.name = "foo" } | rate() by (span.http.url)  | topk(10)

The first part, { resource.service.name = "foo" }, takes all spans in the service foo. The spans are rated by the URL, for example, the most active endpoints on a service.

Adding topk(10) returns the top 10 most common instead of the entire list.

In TraceQL, topk works similar to how it functions in PromQL. The topk function is evaluated at each data point.

If you do a topk of 10, you might get a 20 series. For example, on this data point, the top 10 are A through J.

On the next data point, A through I might still be the top 9, but J might have fallen off for K. Because it’s evaluated at each data point, you’ll get the top series for each data point.

The `bottomk` function

The bottomk function lets you aggregate and process TraceQL metrics by returning only the bottom k results. This function is similar to its equivalent PromQL function.

When a query response is larger than the maximum, you can use bottomk to return only the specified number from 1 through k of the bottom results.

For example:

{ resource.service.name = "foo" } | rate() by (span.http.url)  | bottomk(10)

The first part, { resource.service.name = "foo" }, takes all spans in the service foo. The spans are rated by the URL, for example, the least active endpoints on a service.

Adding bottomk(10) returns the bottom 10 least common instead of the entire list.

In TraceQL, bottomk works similar to how it functions in PromQL. The bottomk function is evaluated at each data point.

If you do a bottomk of 10, you might get a 20 series. For example, on this data point, the bottom 10 are A through J.

On the next data point, A through I might still be the bottom 9, but J might have fallen off for K. Because it’s evaluated at each data point, you’ll get the bottom series for each data point.

Data sampling

TraceQL metrics queries support sampling to optimize performance and control sampling behavior. There are three sampling methods available:

Dynamic sampling using with(sample=true), which automatically determines the optimal sampling strategy and amount based on query characteristics.
Fixed sampling using with(sample=0.xx)
Fixed span sampling using with(span_sample=0.xx), which selects the specified percentage of spans.
Fixed trace sampling using with(trace_sample=0.xx), which selects complete traces for analysis.

Refer to the TraceQL metrics sampling documentation for more information.

Note
Sampling hints only work with TraceQL metrics queries (those using functions like rate(), count_over_time(), etc.).

Dynamic sampling: `with(sample=true)`

Automatically determines optimal sampling strategy based on query selectivity and data volume.

{ resource.service.name="frontend" } | rate() with(sample=true)

Fixed span sampling: `with(span_sample=0.xx)`

Samples a fixed percentage of spans for span-level aggregations.

{ span:status=error } | count_over_time() with(span_sample=0.1)

Fixed trace sampling: `with(trace_sample=0.xx)`

Samples a fixed percentage of traces for trace-level aggregations.

{ } | count() by (resource.service.name) with(trace_sample=0.05)

The `compare` function

Note
This function is primarily used to drive the Comparison tab in Traces Drilldown.

The compare function splits a set of spans into two groups: a selection and a baseline. It returns time-series for all attributes found on the spans to highlight the differences between the two groups.

This powerful function is best understood by using the Comparison tab in Traces Drilldown. You can also under this function by looking at example outputs below.

The function is used like other metrics functions: when it’s placed after any trace query, it converts the query into a metrics query: ...any spanset pipeline... | compare({subset filters}, <topN>, <start timestamp>, <end timestamp>)

Example:

{ resource.service.name="a" && span.http.path="/myapi" } | compare({span:status=error})

This function is generally run as an instant query. An instant query gives a single value at the end of the selected time range. Instant queries are quicker to execute and it often easier to understand their results The returns may exceed gRPC payloads when run as a range query.

Parameters

The compare function has four parameters:

Required. The first parameter is a spanset filter for choosing the subset of spans. This filter is executed against the incoming spans. If it matches, then the span is considered to be part of the selection. Otherwise, it is part of the baseline. Common filters are expected to be things like {span:status=error} (what is different about errors?) or {span:duration>1s} (what is different about slow spans?)
Optional. The second parameter is the top N values to return per attribute. If an attribute exceeds this limit in either the selection group or baseline group, then only the top N values (based on frequency) are returned, and an error indicator for the attribute is included output (see below). Defaults to 10.
Optional. Start and End timestamps in Unix nanoseconds, which can be used to constrain the selection window by time, in addition to the filter. For example, the overall query could cover the past hour, and the selection window only a 5 minute time period in which there was an anomaly. These timestamps must both be given, or neither.

Output

The outputs are flat time-series for each attribute/value found in the spans.

Each series has a label __meta_type which denotes which group it is in, either selection or baseline.

Example output series:

{ __meta_type="baseline", resource.cluster="prod" } 123
{ __meta_type="baseline", resource.cluster="qa" } 124
{ __meta_type="selection", resource.cluster="prod" } 456   <--- significant difference detected
{ __meta_type="selection", resource.cluster="qa" } 125
{ __meta_type="selection", resource.cluster="dev"} 126  <--- cluster=dev was found in the highlighted spans but not in the baseline

When an attribute reaches the topN limit, there will also be present an error indicator. This example means the attribute resource.cluster had too many values.

{ __meta_error="__too_many_values__", resource.cluster=<nil> }

TraceQL metrics on Grafana Labs

Configure TraceQL metrics

Configure TraceQL metrics

Before you begin

Activate and configure the local-blocks processor

Local blocks and metrics-generator in Azure blob storage and Helm

Evaluate query timeouts

Set TraceQL metrics query options

Sampling and performance optimization

TraceQL metrics sampling

TraceQL metrics sampling

Sampling methods

How dynamic sampling works

Before you begin

Dynamic sampling using with(sample=true)

Fixed span sampling using with(span_sample=0.xx)

Fixed trace sampling using with(trace_sample=0.xx)

TraceQL metrics functions

TraceQL metrics functions

Available functions

Group results with by()

The rate function

The count_over_time function

The step parameter

Over-time functions

The sum_over_time function

The min_over_time function

The max_over_time function

The avg_over_time function

The quantile_over_time function

The histogram_over_time function

Multi-stage metrics queries

The topk function

The bottomk function

Data sampling

Dynamic sampling: with(sample=true)

Fixed span sampling: with(span_sample=0.xx)

Fixed trace sampling: with(trace_sample=0.xx)

The compare function

Parameters

Output

Activate and configure the `local-blocks` processor

Dynamic sampling using `with(sample=true)`

Fixed span sampling using `with(span_sample=0.xx)`

Fixed trace sampling using `with(trace_sample=0.xx)`

Group results with `by()`

The `rate` function

The `count_over_time` function

The `step` parameter

The `sum_over_time` function

The `min_over_time` function

The `max_over_time` function

The `avg_over_time` function

The `quantile_over_time` function

The `histogram_over_time` function

The `topk` function

The `bottomk` function

Dynamic sampling: `with(sample=true)`

Fixed span sampling: `with(span_sample=0.xx)`

Fixed trace sampling: `with(trace_sample=0.xx)`

The `compare` function