Metrics from traces on Grafana Labs

TraceQL metrics

Thu, 28 May 2026 17:50:33 +0100

TraceQL metrics

TraceQL metrics is a feature in Grafana Tempo that creates metrics from traces.

Metric queries extend trace queries by applying a function to trace query results. This powerful feature allows for ad hoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs.

Traces are a unique observability signal that contain causal relationships between the components in your system.

TraceQL metrics can help answer questions like this:

How many database calls across all systems are downstream of your application?
What services beneath a given endpoint are failing?
What services beneath an endpoint are slow?

TraceQL metrics can help you answer these questions by parsing your traces in aggregate.

TraceQL metrics are powered by the TraceQL metrics API.

RED metrics, TraceQL, and PromQL

RED is an acronym for three types of metrics:

Rate, the number of requests per second
Errors, the number of those requests that are failing
Duration, the amount of time those requests take

For more information about the RED method, refer to The RED Method: how to instrument your services.

You can write TraceQL metrics queries to compute rate, errors, and durations over different groups of spans.

For more information on how to use TraceQL metrics to investigate issues, refer to Solve problems with metrics queries.

Enable and use TraceQL metrics

To use TraceQL metrics, you need to enable them on your Tempo database. Refer to Configure TraceQL metrics for more information.

From there, you can either query the TraceQL metrics API directly (for example, with curl) or using Grafana (recommended). To run TraceQL metrics queries in Grafana, you need Grafana Cloud or Grafana 10.4 or later. No extra configuration is needed. Use a Tempo data source that points to a Tempo database with TraceQL metrics enabled.

Refer to Solve problems using metrics queries for some real-world examples.

Functions

TraceQL metrics queries currently include the following functions for aggregating over groups of spans: rate, count_over_time, sum_over_time, max_over_time, min_over_time, avg_over_time, quantile_over_time, histogram_over_time, and compare. These functions can be added as an operator at the end of any TraceQL query.

The topk and bottomk functions and comparison operators (>, >=, <, <=, =, !=) are supported on TraceQL metrics results.

For detailed information and example queries for each function, refer to TraceQL metrics functions.

Exemplars

Exemplars are a powerful feature of TraceQL metrics. They allow you to see an exact trace that contributed to a given metric value. This is particularly useful when you want to understand why a given metric is high or low.

Exemplars are available in TraceQL metrics for all range queries. To get exemplars, you need to configure it in the query-frontend with the parameter query_frontend.metrics.max_exemplars, or pass a query hint in your query.

Example:

{ span:name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target) with (exemplars=true)

Faster read path (experimental)

Note
Tempo is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the feature toggle in Grafana to use this feature. Do not enable this feature in production environments.

In vParquet5, you can use an experimental span-only fetch layer to significantly improve performance for most metrics queries. This optimized read path processes individual spans instead of full traces, reducing latency and memory usage.

You must enable the faster read path explicitly using a query hint or a per-tenant override. Once enabled, it applies to metrics queries that don’t require knowledge of the full trace structure. Queries using structural operators like >>, <<, ~, !>>, !<<, or !~ still use the standard fetch layer.

Enable with a query hint

Add the spanonly_fetch=true hint to your query. This hint requires unsafe_query_hints to be enabled for the tenant.

{ resource.service.name = "frontend" } | rate() by (status) with (spanonly_fetch=true)

Enable with a per-tenant override

Operators can enable the faster read path by default for a tenant using the metrics_spanonly_fetch override. When set, it applies to all eligible metrics queries for that tenant without requiring a query hint. This override doesn’t require unsafe_query_hints.

overrides:
  'tenant-id':
    read:
      metrics_spanonly_fetch: true

When unsafe_query_hints is also enabled for the tenant, the spanonly_fetch query hint takes precedence over the per-tenant override. Users can set spanonly_fetch=false to opt out, or spanonly_fetch=true to opt in even when the override is disabled.

Metrics-generator

Thu, 28 May 2026 17:50:33 +0100

Metrics-generator

Metrics-generator is an optional Tempo component that derives metrics from ingested traces. The metrics-generator consumes trace data from Kafka and writes metrics to a Prometheus data source using the Prometheus remote-write protocol.

Architecture

Metrics-generator consumes trace data from Kafka to generate metrics from traces.

The metrics-generator internally runs a set of processors. Each processor ingests spans and produces metrics. Every processor derives different metrics. Currently, the following processors are available:

Service graphs
Span metrics
Host info

Note
Tempo 3.0 removed the local-blocks processor. Remove any local-blocks entries from your metrics_generator.processors configuration. The live-store component handles TraceQL metrics queries on recent data.

Instrumented applications send traces to the distributor, which writes them to Kafka. The metrics-generator consumes trace data from Kafka and runs it through its configured processors (span metrics, service graphs, and host info). Each processor derives a different set of metrics, which the metrics-generator then remote-writes to a Prometheus-compatible backend such as Prometheus or Grafana Mimir.

Service graphs

Service graphs are the representations of the relationships between services within a distributed system.

This service graphs processor builds a map of services by analyzing traces, with the objective to find edges. Edges are spans with a parent-child relationship, that represent a jump (for example, a request) between two services. The processor records the request count and duration as metrics and uses them to represent the graph.

To learn more about this processor, refer to the service graph documentation.

Span metrics

The span metrics processor derives RED (Request, Error, and Duration) metrics from spans.

The span metrics processor computes the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any tag or attribute present in the span. The more dimensions you enable, the higher the cardinality of the generated metrics.

To learn more about this processor, refer to the span metrics documentation.

Host info

The host info processor emits a traces_host_info gauge metric derived from resource attributes on incoming spans. It identifies hosts sending traces using configurable identifiers (by default, k8s.node.name and host.id) and produces labels grafana_host_id and host_source. This processor is useful for correlating trace data with host-level infrastructure metrics.

To learn more about the configuration, refer to the Metrics-generator section of the Tempo Configuration documentation.

Remote writing metrics

The metrics-generator runs a Prometheus Agent that periodically sends metrics to a remote_write endpoint. The remote_write endpoint is configurable and can be any Prometheus-compatible endpoint. To learn more about the endpoint configuration, refer to the Metrics-generator section of the Tempo Configuration documentation. Use metrics_generator.registry.collection_interval to control the writing interval.

When you enable multi-tenancy, the metrics-generator forwards the X-Scope-OrgID header of the original request to the remote_write endpoint. To disable this behavior, set remote_write_add_org_id_header to false.

Native histograms

Native histograms are a data type in Prometheus that can produce, store, and query high-resolution histograms of observations. It usually offers higher resolution and more straightforward instrumentation than classic histograms.

The metrics-generator supports the ability to produce native histograms for high-resolution data. Users must update the receiving endpoint to ingest native histograms, and update histogram queries in their dashboards.

To learn more about the configuration, refer to the Metrics-generator section of the Tempo Configuration documentation.

Use metrics-generator in Grafana Cloud

If you want to enable metrics-generator for your Grafana Cloud account, refer to the Metrics-generator in Grafana Cloud documentation.

Enabling metrics generation and remote writing them to Grafana Cloud Metrics produces extra active series that could impact your billing. For more information on billing, refer to Understand your invoice.

Partition handoff during rollouts

The metrics-generator uses Kafka static membership (instance_id) so that a pod restarting with the same identity can rejoin the consumer group immediately (without waiting for the session timeout) and reclaim its previously held partitions. This is the right behaviour for StatefulSet deployments, where pod names are stable across restarts (for example, metrics-generator-0).

For Deployment deployments, pod names change on every restart (for example, metrics-generator-7f9d4b6c5-xk2pj). Because the new pod has a different name, it registers as a new static member, and the old member slot holds its partitions until the session timeout expires (typically several minutes). During that window the partitions are idle and no metrics are generated from the corresponding trace data.

To avoid this delay, set leave_consumer_group_on_shutdown: true on the metrics-generator. When enabled, the shutting-down pod explicitly sends a LeaveGroup request to the Kafka coordinator before closing. The coordinator can then reassign the partitions immediately to the new pod, rather than waiting for the session timeout.

metrics_generator:
  leave_consumer_group_on_shutdown: true  # recommended for Deployment rollouts

Set leave_consumer_group_on_shutdown: false (the default) for StatefulSet deployments. With a stable instance_id, sending LeaveGroup on shutdown would cause an unnecessary leave-and-rejoin rebalance on every restart, briefly interrupting metrics generation for all tenants sharing the same consumer group.

Note
leave_consumer_group_on_shutdown requires Kafka 2.4 or later, which introduced the per-member InstanceID field in the LeaveGroup request (KIP-345).

Multitenancy

Tempo supports multitenancy in the metrics-generator through the use of environment variables and per-tenant overrides. Refer to the Multitenant Support for Metrics-Generator documentation for more information.

Span metrics

Thu, 28 May 2026 17:50:33 +0100

Span metrics

Span metrics are generated from traces and can be used to create service graphs. You can create span metrics by enabling the feature in metrics-generator or using Grafana Alloy.

Use the span metrics processor
The span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.
Use Alloy to generate span metrics from spans
Span metrics allow you to generate metrics from your tracing data automatically.

Service graphs

Thu, 28 May 2026 17:50:33 +0100

Service graphs

A service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components:

Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs help you to understand the structure of the system.
Provide a high-level overview of the health of your system. Service graphs display error rates, latencies, as well as other relevant data.
Provide an historic view of a system’s topology. Distributed systems change very frequently, and service graphs offer a way of seeing how these systems have evolved over time.

Service graphs can be generated from metrics created by the metrics-generator or Grafana Alloy. Refer to Enable service graphs for more information on how to enable service graphs in Tempo.

How they work

The metrics-generator and Grafana Alloy both process traces and generate service graphs in the form of Prometheus metrics.

Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request. The processor uses the OpenTelemetry semantic conventions to detect a myriad of requests.

It supports the following requests:

A direct request between two services where the outgoing and the incoming span must have span.kind, client, and server, respectively.
A request across a messaging system where the outgoing and the incoming span must have span.kind, producer, and consumer respectively.
A database request; in this case the processor looks for spans containing attributes span.kind=client as well as one of db.namespace, db.name or db.system. You can customize which attributes identify a database span using the database_name_attributes configuration option. See below for how the name of the node is determined for a database request.

The processor keeps every span that can form a request pair in an in-memory store until the corresponding pair span arrives or the maximum waiting time passes. When either condition occurs, the processor records the request and removes it from the local store.

Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.

  traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20

Virtual nodes

Virtual nodes are nodes that form part of the lifecycle of a trace, but spans for them aren’t collected because they’re outside the user’s reach or aren’t instrumented. For example, you might not collect spans for an external service for payment processing that’s outside user interaction.

The processor detects virtual nodes in two ways:

Uninstrumented client (missing client span): The root span has span.kind set to server or consumer, with no matching client span. This indicates that the request or message was initiated by an external system that isn’t instrumented, like a scheduler, a frontend application, or an engineer using curl.
- In the Tempo metrics-generator, the processor checks the configured peer_attributes on the server span first. If it finds a matching attribute, it uses that value as the client node name. Otherwise, the client node name defaults to user.
- In Grafana Alloy and the OpenTelemetry Collector servicegraph connector, the connector doesn’t evaluate peer attributes for this case. The client node name always defaults to user and you can’t override it. An upstream feature request exists to add this capability.
Uninstrumented server (missing server span): A client span doesn’t have its matching server span, but has a peer attribute present. In this case, the client called an external service that doesn’t send spans. The processor uses the peer attribute value as the virtual server node name.
- The default peer attributes are peer.service, db.name, and db.system.
- The processor searches the attributes in order and uses the first match as the virtual node name.

The processor identifies a database node when the span has at least one db.namespace, db.name, or db.system attribute.

The processor determines the database node name using the following span attributes in order of precedence: peer.service, server.address, network.peer.address:network.peer.port, db.namespace, db.name.

Metrics

The following metrics are exported:

Metric	Type	Labels	Description
`traces_service_graph_request_total`	Counter	client, server, connection_type	Total count of requests between two nodes
`traces_service_graph_request_failed_total`	Counter	client, server, connection_type	Total count of failed requests between two nodes
`traces_service_graph_request_server_seconds`	Histogram	client, server, connection_type	Time for a request between two nodes as seen from the server
`traces_service_graph_request_client_seconds`	Histogram	client, server, connection_type	Time for a request between two nodes as seen from the client
`traces_service_graph_request_messaging_system_seconds`	Histogram	client, server, connection_type	(Off by default) Time between publisher and consumer for services communicating through a messaging system
`traces_service_graph_unpaired_spans_total`	Counter	client, server, connection_type	Total count of unpaired spans
`traces_service_graph_dropped_spans_total`	Counter	client, server, connection_type	Total count of dropped spans

The processor measures duration from both the client and server sides.

Possible values for connection_type: unset, virtual_node, messaging_system, or database.

You can include additional labels using the dimensions configuration option or the enable_virtual_node_label option.

Duplicate dimensions are allowed after Prometheus label name conversion. This supports environments where different instrumentation libraries use different attribute naming conventions, for example, deployment.environment and deployment_environment. When a collision occurs, the last configured value wins.

Since the service graph processor has to process both sides of an edge, it needs to process all spans of a trace to function properly. If spans of a trace spread across multiple instances, the processor can’t pair them reliably.

Activate `enable_virtual_node_label`

Activating this feature adds the following label and corresponding values:

Label	Possible Values	Description
`virtual_node`	`unset`, `client`, `server`	Explicitly indicates the uninstrumented side

Configuration options

The service graphs processor has several configuration options beyond dimensions and enable_virtual_node_label. For the full YAML schema and defaults, refer to the configuration reference.

Span multiplier

When traces are sampled, the raw request counts produced by the service graph processor underrepresent actual traffic. The span_multiplier_key option specifies a span or resource attribute that contains the sampling ratio. The processor computes the inverse of this value to scale the metrics accordingly. For example, if a span has attribute X-SampleRatio=0.1 (10% sampling), setting span_multiplier_key: "X-SampleRatio" causes each sampled span to count as 10 requests.

The enable_tracestate_span_multiplier option provides an alternative approach that extracts the multiplier from the W3C tracestate header using the OpenTelemetry probability sampling threshold (ot=th:<hex>). When enabled, the tracestate threshold takes priority over span_multiplier_key.

Database name attributes

The database_name_attributes option controls which span attributes the processor uses to identify a span as a database request. The defaults are db.namespace, db.name, and db.system. You can override this list to match your instrumentation if it uses non-standard attribute names.

Filter policies

The filter_policies option lets you include or exclude spans from service graph generation. It uses the same policy format as span metrics.

The service graph processor only evaluates spans that can form edges: SPAN_KIND_CLIENT, SPAN_KIND_SERVER, SPAN_KIND_PRODUCER, and SPAN_KIND_CONSUMER. When one side of an edge is filtered out, Tempo keeps a best-effort marker for the dropped side and drops the matching side if it arrives later or is already buffered. This reduces skewed edges and unwanted virtual nodes. The marker cache uses wait as its TTL and max_items as its maximum size.

Policy behavior is:

include: all include policies must match.
include_any: any include_any policy can match. If only include_any policies are configured, non-matching spans are excluded.
exclude: matching spans are rejected, even if they match an include rule.

Use scoped keys for resource and span attributes, such as resource.service.name or span.http.route. The supported intrinsic keys are name, status, and kind. Supported attribute value types are bool, double, int, and string.

This example excludes service graph spans from shop-backend:

metrics_generator:
  processor:
    service_graphs:
      filter_policies:
        - exclude:
            match_type: strict
            attributes:
              - key: resource.service.name
                value: shop-backend

Monitor service graph filtering with:

tempo_metrics_generator_spans_discarded_total{reason="service_graphs_filtered", processor="service-graphs"}
tempo_metrics_generator_processor_service_graphs_dropped_edges_total
tempo_metrics_generator_processor_service_graphs_dropped_span_side_cache_overflow_total

Metrics from traces on Grafana Labs

TraceQL metrics

TraceQL metrics

RED metrics, TraceQL, and PromQL

Enable and use TraceQL metrics

Functions

Exemplars

Faster read path (experimental)

Enable with a query hint

Enable with a per-tenant override

Metrics-generator

Metrics-generator

Architecture

Service graphs

Span metrics

Host info

Remote writing metrics

Native histograms

Use metrics-generator in Grafana Cloud

Partition handoff during rollouts

Multitenancy

Span metrics

Span metrics

Service graphs

Service graphs

How they work

Virtual nodes

Metrics

Activate enable_virtual_node_label

Configuration options

Span multiplier

Database name attributes

Filter policies

Activate `enable_virtual_node_label`