Tempo documentation Metrics-generator Generate metrics from spans
Open source

Generate metrics from spans

The span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics.

Span metrics generate two metrics:

  • A counter that computes requests
  • A histogram that tracks the distribution of durations of all requests

Span metrics are of particular interest if your system is not monitored with metrics, but it has distributed tracing implemented. You get out-of-the-box metrics from your tracing pipeline.

Even if you already have metrics, span metrics can provide in-depth monitoring of your system. The generated metrics will show application level insight into your monitoring, as far as tracing gets propagated through your applications.

Last but not least, span metrics lower the entry barrier for using exemplars. An exemplar is a specific trace representative of measurement taken in a given time interval. Since traces and metrics co-exist in the metrics-generator, exemplars can be automatically added, providing additional value to these metrics.

How to run

To enable service graphs in Tempo/GET, enable the metrics generator and add an overrides section which enables the span-metrics generator. See here for configuration details.

How it works

The span metrics processor works by inspecting every received span and computing the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any attribute present in the span.

This processor is designed with the goal to mirror the implementation from the OpenTelemetry Collector of the processor with the same name.


The following metrics are exported:

traces_spanmetrics_latencyHistogramDimensionsDuration of the span
traces_spanmetrics_calls_totalCounterDimensionsTotal count of the span
traces_spanmetrics_size_totalCounterDimensionsTotal size of spans ingested
Note: In Tempo 1.4 and 1.4.1, the histogram metric was called traces_spanmetrics_duration_seconds. This was changed later to be consistent with the metrics generated by the Grafana Agent and the OpenTelemetry Collector.

By default, the metrics processor adds the following labels to each metric: service, span_name, span_kind, status_code, status_message. Additional user defined labels can be created using the dimensions configuration option. When a configured dimension collides with one of the default labels (e.g. status_code), the label for the respective dimension is prefixed with double underscore (i.e. __status_code).

If you use ratio based sampler you can use custom sampler below to not lose metric information, you also need to set metrics_generator.processor.span_metrics.span_multiplier_key to "X-SampleRatio"

package tracer
import (
	tracesdk ""

type RatioBasedSampler struct {
	innerSampler        tracesdk.Sampler
	sampleRateAttribute attribute.KeyValue

func NewRatioBasedSampler(fraction float64) RatioBasedSampler {
	innerSampler := tracesdk.TraceIDRatioBased(fraction)
	return RatioBasedSampler{
		innerSampler:        innerSampler,
		sampleRateAttribute: attribute.Float64("X-SampleRatio", fraction),

func (ds RatioBasedSampler) ShouldSample(parameters tracesdk.SamplingParameters) tracesdk.SamplingResult {
	sampler := ds.innerSampler
	result := sampler.ShouldSample(parameters)
	if result.Decision == tracesdk.RecordAndSample {
		result.Attributes = append(result.Attributes, ds.sampleRateAttribute)
	return result

func (ds RatioBasedSampler) Description() string {
	return "Ratio Based Sampler which gives information about sampling ratio"


Span metrics overview