Metrics from traces on Grafana Labs

TraceQL metrics

Thu, 09 Apr 2026 14:59:14 +0000

TraceQL metrics

Caution
TraceQL metrics is an public preview feature. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available TraceQL metrics are enabled by default in Grafana Cloud.

TraceQL metrics is a feature in Grafana Tempo that creates metrics from traces.

Metric queries extend trace queries by applying a function to trace query results. This powerful feature allows for ad hoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs.

Traces are a unique observability signal that contain causal relationships between the components in your system.

TraceQL metrics can help answer questions like this:

How many database calls across all systems are downstream of your application?
What services beneath a given endpoint are failing?
What services beneath an endpoint are slow?

TraceQL metrics can help you answer these questions by parsing your traces in aggregate.

TraceQL metrics are powered by the TraceQL metrics API.

RED metrics, TraceQL, and PromQL

RED is an acronym for three types of metrics:

Rate, the number of requests per second
Errors, the number of those requests that are failing
Duration, the amount of time those requests take

For more information about the RED method, refer to The RED Method: how to instrument your services.

You can write TraceQL metrics queries to compute rate, errors, and durations over different groups of spans.

For more information on how to use TraceQL metrics to investigate issues, refer to Solve problems with metrics queries.

Enable and use TraceQL metrics

To use TraceQL metrics, you need to enable them on your Tempo database. Refer to Configure TraceQL metrics for more information.

From there, you can either query the TraceQL metrics API directly (for example, with curl) or using Grafana (recommended). To run TraceQL metrics queries in Grafana, you need Grafana Cloud or Grafana 10.4 or later. No extra configuration is needed. Use a Tempo data source that points to a Tempo database with TraceQL metrics enabled.

Refer to Solve problems using metrics queries for some real-world examples.

Functions

TraceQL metrics queries currently include the following functions for aggregating over groups of spans: rate, count_over_time, sum_over_time, max_over_time, min_over_time, avg_over_time, quantile_over_time, histogram_over_time, and compare. These functions can be added as an operator at the end of any TraceQL query.

The topk and bottomk functions are supported on TraceQL metrics functions.

For detailed information and example queries for each function, refer to TraceQL metrics functions.

Exemplars

Exemplars are a powerful feature of TraceQL metrics. They allow you to see an exact trace that contributed to a given metric value. This is particularly useful when you want to understand why a given metric is high or low.

Exemplars are available in TraceQL metrics for all range queries. To get exemplars, you need to configure it in the query-frontend with the parameter query_frontend.metrics.max_exemplars, or pass a query hint in your query.

Example:

{ span:name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target) with (exemplars=true)

Metrics-generator

Thu, 09 Apr 2026 14:59:14 +0000

Metrics-generator

Metrics-generator is an optional Tempo component that derives metrics from ingested traces. If present, the distributor writes received spans to both the ingester and the metrics-generator. The metrics-generator processes spans and writes metrics to a Prometheus data source using the Prometheus remote write protocol.

Architecture

Metrics-generator leverages the data available in the ingest path in Tempo to provide additional value by generating metrics from traces.

The metrics-generator internally runs a set of processors. Each processor ingests spans and produces metrics. Every processor derives different metrics. Currently, the following processors are available:

Service graphs
Span metrics
Local blocks

Service graphs

Service graphs are the representations of the relationships between services within a distributed system.

This service graphs processor builds a map of services by analyzing traces, with the objective to find edges. Edges are spans with a parent-child relationship, that represent a jump (for example, a request) between two services. The amount of request and their duration are recorded as metrics, which are used to represent the graph.

To learn more about this processor, refer to the service graph documentation.

Span metrics

The span metrics processor derives RED (Request, Error, and Duration) metrics from spans.

The span metrics processor computes the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any tag or attribute present in the span. The more dimensions are enabled, the higher the cardinality of the generated metrics.

To learn more about this processor, refer to the span metrics documentation.

Local blocks

The local blocks processor stores spans for a set period of time and enables more complex APIs to perform calculations on the data. The processor must be enabled for certain metrics APIs to function.

Remote writing metrics

The metrics-generator runs a Prometheus Agent that periodically sends metrics to a remote_write endpoint. The remote_write endpoint is configurable and can be any Prometheus-compatible endpoint. To learn more about the endpoint configuration, refer to the Metrics-generator section of the Tempo Configuration documentation. Writing interval can be controlled via metrics_generator.registry.collection_interval.

When multi-tenancy is enabled, the metrics-generator forwards the X-Scope-OrgID header of the original request to the remote_write endpoint. This feature can be disabled by setting remote_write_add_org_id_header to false.

Native histograms

Native histograms are a data type in Prometheus that can produce, store, and query high-resolution histograms of observations. It usually offers higher resolution and more straightforward instrumentation than classic histograms.

The metrics-generator supports the ability to produce native histograms for high-resolution data. Users must update the receiving endpoint to ingest native histograms, and update histogram queries in their dashboards.

To learn more about the configuration, refer to the Metrics-generator section of the Tempo Configuration documentation.

Use metrics-generator in Grafana Cloud

If you want to enable metrics-generator for your Grafana Cloud account, refer to the Metrics-generator in Grafana Cloud documentation.

Enabling metrics generation and remote writing them to Grafana Cloud Metrics produces extra active series that could impact your billing. For more information on billing, refer to Understand your invoice.

Multitenancy

Tempo supports multitenancy in the metrics-generator through the use of environment variables and per-tenant overrides. Refer to the Multitenant Support for Metrics-Generator documentation for more information.

Span metrics

Thu, 09 Apr 2026 14:59:14 +0000

Span metrics

Span metrics are generated from traces and can be used to create service graphs. You can create span metrics by enabling the feature in metrics-generator or using Grafana Alloy.

Use the span metrics processor
The span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.
Use Alloy to generate span metrics from spans
Span metrics allow you to generate metrics from your tracing data automatically.

Service graphs

Thu, 09 Apr 2026 14:59:14 +0000

Service graphs

A service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components:

Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs help you to understand the structure of the system.
Provide a high-level overview of the health of your system. Service graphs display error rates, latencies, as well as other relevant data.
Provide an historic view of a system’s topology. Distributed systems change very frequently, and service graphs offer a way of seeing how these systems have evolved over time.

Service graphs can be generated from metrics created by the metrics-generator or Grafana Alloy. Refer to Enable service graphs for more information on how to enable service graphs in Tempo.

How they work

The metrics-generator and Grafana Alloy both process traces and generate service graphs in the form of Prometheus metrics.

Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request. The processor uses the OpenTelemetry semantic conventions to detect a myriad of requests.

It supports the following requests:

A direct request between two services where the outgoing and the incoming span must have span.kind, client, and server, respectively.
A request across a messaging system where the outgoing and the incoming span must have span.kind, producer, and consumer respectively.
A database request; in this case the processor looks for spans containing attributes span.kind=client as well as one of db.namespace, db.name or db.system. See below for how the name of the node is determined for a database request.

The processor keeps every span that can form a request pair in an in-memory store until the corresponding pair span arrives or the maximum waiting time passes. When either condition occurs, the processor records the request and removes it from the local store.

Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.

  traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20

Virtual nodes

Virtual nodes are nodes that form part of the lifecycle of a trace, but spans for them aren’t collected because they’re outside the user’s reach or aren’t instrumented. For example, you might not collect spans for an external service for payment processing that’s outside user interaction.

The processor detects virtual nodes in two ways:

Uninstrumented client (missing client span): The root span has span.kind set to server or consumer, with no matching client span. This indicates that the request or message was initiated by an external system that isn’t instrumented, like a scheduler, a frontend application, or an engineer using curl.
- In the Tempo metrics-generator, the processor checks the configured peer_attributes on the server span first. If it finds a matching attribute, it uses that value as the client node name. Otherwise, the client node name defaults to user.
- In Grafana Alloy and the OpenTelemetry Collector servicegraph connector, the connector doesn’t evaluate peer attributes for this case. The client node name always defaults to user and you can’t override it. An upstream feature request exists to add this capability.
Uninstrumented server (missing server span): A client span doesn’t have its matching server span, but has a peer attribute present. In this case, the client called an external service that doesn’t send spans. The processor uses the peer attribute value as the virtual server node name.
- The default peer attributes are peer.service, db.name, and db.system.
- The processor searches the attributes in order and uses the first match as the virtual node name.

The processor identifies a database node when the span has at least one db.namespace, db.name, or db.system attribute.

The processor determines the database node name using the following span attributes in order of precedence: peer.service, server.address, network.peer.address:network.peer.port, db.namespace, db.name.

Metrics

The following metrics are exported:

Metric	Type	Labels	Description
`traces_service_graph_request_total`	Counter	client, server, connection_type	Total count of requests between two nodes
`traces_service_graph_request_failed_total`	Counter	client, server, connection_type	Total count of failed requests between two nodes
`traces_service_graph_request_server_seconds`	Histogram	client, server, connection_type	Time for a request between two nodes as seen from the server
`traces_service_graph_request_client_seconds`	Histogram	client, server, connection_type	Time for a request between two nodes as seen from the client
`traces_service_graph_request_messaging_system_seconds`	Histogram	client, server, connection_type	(Off by default) Time between publisher and consumer for services communicating through a messaging system
`traces_service_graph_unpaired_spans_total`	Counter	client, server, connection_type	Total count of unpaired spans
`traces_service_graph_dropped_spans_total`	Counter	client, server, connection_type	Total count of dropped spans

The processor measures duration from both the client and server sides.

Possible values for connection_type: unset, virtual_node, messaging_system, or database.

You can include additional labels using the dimensions configuration option or the enable_virtual_node_label option.

Since the service graph processor has to process both sides of an edge, it needs to process all spans of a trace to function properly. If spans of a trace spread across multiple instances, the processor can’t pair them reliably.

Activate `enable_virtual_node_label`

Activating this feature adds the following label and corresponding values:

Label	Possible Values	Description
`virtual_node`	`unset`, `client`, `server`	Explicitly indicates the uninstrumented side

Metrics from traces on Grafana Labs

TraceQL metrics

TraceQL metrics

RED metrics, TraceQL, and PromQL

Enable and use TraceQL metrics

Functions

Exemplars

Metrics-generator

Metrics-generator

Architecture

Service graphs

Span metrics

Local blocks

Remote writing metrics

Native histograms

Use metrics-generator in Grafana Cloud

Multitenancy

Span metrics

Span metrics

Service graphs

Service graphs

How they work

Virtual nodes

Metrics

Activate enable_virtual_node_label

Activate `enable_virtual_node_label`