Metrics-generator on Grafana Labs

Active series

Thu, 28 May 2026 17:50:33 +0100

Active series

An active series is a time series that receives new data points or samples. When you stop writing new data points to a time series, shortly afterwards it’s no longer considered active.

Metrics generated by the metrics-generator can provide both RED (Rate/Error/Duration) metrics and interdependency graphs between services in a trace (the Service Graph functionality in Grafana). These capabilities rely on a set of generated span metrics and service metrics.

Any spans that are ingested by Tempo can create many metric series. However, this doesn’t mean that every time a span is ingested that a new active series is created.

The number of active series generated depends on the label pairs generated from span data that are associated with the metrics, similar to other Prometheus-formatted data.

For additional information, refer to the Active series and DPM documentation.

Active series calculation

Active series for a metric increase when a new value for a label key is introduced. For example, the span_kind label has a total of five possible values, and the status_code label has a total of three possible values.

At first glance, you might make an assumption that this means that at least 15 (5*3) active series will be generated for each span. But this isn’t the case.

Let’s consider a span that’s emitted from some piece of code in a service:

Here’s a single service with a single span. If the code inside the span never leaves the service, then the span_kind label generated by the metrics generator will be SPAN_KIND_INTERNAL and never deviate. It’ll never be one of the other four possible values.

Similarly, if the code inside the span never errors, it’ll only have the STATUS_CODE_OK state for the span_status label. This means that the metrics generator will only generate a single active series, where the service name will be Service 1 and the span name will be span1. If we looked at the Prometheus data for the traces_spanmetrics_call_total metric, we’d see a single active series:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	1

It doesn’t matter how many times that span occurs in a trace either, for example maybe a span is generated within a loop. In code run once, 10 times, 100 times, 1000 times, only a single active series will be produced, where a counter might be increased 1, 10, 100, or 1000 times:

If you looked at the Prometheus data, you’d see an instant value for traces_spanmetrics_call_total similar to the table. Again, one active series for the metric:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	120

However, let’s now assume that it does loop and there are occasionally errors.

There are now two potential outcomes for a span when the code loops: one where everything successfully completes and one where there is an error. This means that when the span completes status_code is now either STATUS_CODE_OK or STATUS_CODE_ERROR. Because of that, the label values can be one of two values on a metric, and we now have two active series being generated based on the status_code, one for the OK status and one for the error.

Again, we could loop once, 10 times, 100, or more times, but there will only ever be two active series.

If we now looked at Prometheus instant values for traces_spanmetrics_call_total, we’d now see the following table:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	96
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	24

What happens if you call out to another service though? Let’s add an option where, based on some arbitrary data, we sometimes make a downstream call to another service, but otherwise continue to runs loops in our own service:

In this scenario, span1’s span_kind label would now be one of either SPAN_KIND_INTERNAL or SPAN_KIND_CLIENT (as it has acted as a client calling a downstream server). If a call to the downstream service could also potentially fail, then for SPAN_KIND_CLIENT, the status_code could be either STATUS_CODE_ERROR or STATUS_CODE_OK.

At this point, traces_spanmetrics_call_total would have four different variations in labels:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	34
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	6
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_OK	23
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_ERROR	3

Because of the variation in values, we now have four active series for our metric instead of one. But, as far as Service 1 is concerned, there’s still only four active series, because there isn’t any other variation of the values for labels. You can run 1 trace, 10 traces, 100 traces (each with however many loops of spans there are) and only four active series will ever be produced.

We’ve actually only told half the story in our last diagram. Service 1 called a second service, Service 2, which continues the trace by adding a new span, span2. If there was a loop inside Service 2 with a single span that was generated from an upstream call from Service 1, and then a number of spans that were driven internally, which could also error, we’d end up with the possible values in the metric for traces_spanmetrics_call_total below:

service	span_name	span_kind	status_code	Metric value
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_OK	89
Service 1	span1	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	13
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_OK	44
Service 1	span1	SPAN_KIND_CLIENT	STATUS_CODE_ERROR	9
Service 2	span2	SPAN_KIND_SERVER	STATUS_CODE_OK	30
Service 2	span2	SPAN_KIND_SERVER	STATUS_CODE_ERROR	14
Service 2	span2	SPAN_KIND_INTERNAL	STATUS_CODE_OK	99
Service 2	span2	SPAN_KIND_INTERNAL	STATUS_CODE_ERROR	23

At this point, all our traces will be composed of two potential span names, each of which produce two separate types of span_kind and two separate types of status_code. So we have eight active series for a metric.

The variability of values for each potential span condition determines the number of active series being produced by Tempo when ingesting spans for a trace, and not the number of traces of spans that are seen.

Custom span attributes

There’s another consideration for active series: extra label key/value pairs that can be added onto metrics from a span’s attributes. The Tempo metrics generator allows the user to use arbitrary span attributes to be created as label pairs for metrics. When considering the number of active series generated, you also need to determine how many possible values there are for the span attribute being turned into a label.

For example, if you added an http.method span attribute into a metric label pair, there are five possible values (because there are five possible REST methods):

HEAD
GET
POST
PUT
DELETE

If this label pair is added to every span metric, that’s another 5 potential active series generated for each metric (in all likelihood this is a very worst case scenario, very few spans will call all five REST methods). Instead of 8 active series in the last table above, we’d have 40 (8 * 5).

Cardinality

Thu, 28 May 2026 17:50:33 +0100

Cardinality

Cardinality refers to the total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate. For more information on cardinality, refer to the What are cardinality spikes and why do they matter? blog post.

Because writes to a time-series database (TSDB) database are in series, high cardinality doesn’t make a big difference to performance at ingest. However, cardinality can have a major impact on querying where, the higher the cardinality, the more items are required to be iterated over.

Traces collection and metrics

Tempo’s server-side metrics generation adds functionality to the collection of traces by creating Prometheus-based metrics that track a variety of metrics such as:

Total span call counts
Span latency histograms
Total span size count

The metrics-generator creates metrics which define the relationship between services via edges and nodes. Each of these metrics are queryable using a set of Prometheus labels (key/value pairs).

Each new value for a label increases the number of active series associated with a metric. (To learn more about active series, refer to the Trace active series documentation.)

This is also known as an increase in cardinality, and the number of active series generated for a metric is directly proportional to the number of labels that exist for that metrics alongside the number of values each label has added.

In a non-modified instance of the metrics generator, a small number of labels are added automatically. Because labels like span_kind and status_code only have a few valid values, the largest variable for the number of active series produced for each metric depends on the number of service names and span names associated with trace spans.

The metrics-generator can also be configured to also add extra labels on metrics, using span attribute key/value pairs which are mapped directly to these labels. Refer to the custom span attribute documentation for more information.

Be careful when configuring custom attributes: the greater the number of values seen in a specific attribute, the greater the number of active series are produced. For more information about active series, refer to the active series documentation

Let’s say that you are adding a custom attribute that includes unique customer IDs as a metrics label. If you have 100 customers, this could potentially multiply the number of active series generated by up to 100 (for example, going from 25,000 active series to 2.5M). Always consider which attributes are useful as labels for querying metrics, as well as the cardinality that they can increase metrics by.

Dry-running the metrics-generator

A good practice, before fully enabling metrics-generator, is to run the metrics-generator in dry-run mode. Using the dry-run mode generates metrics but doesn’t collect them, and thus doesn’t write them to a metrics storage database. The override metrics_generator.disable_collection is defined for this use-case.

To get an estimate, set the override to true, and run the metrics-generator as you would normally. Then, check tempo_metrics_generator_registry_active_series for the calculated estimation of active series for your set-up.

If your active series limit is already reached and tempo_metrics_generator_registry_active_series no longer reflects true demand, use the tempo_metrics_generator_registry_active_series_demand_estimate metric instead. This metric uses HyperLogLog estimation to approximate the actual cardinality even when limits are in effect.

Reduce cardinality with span name sanitization

The span_name label is one of the largest contributors to high cardinality in generated metrics. Applications that include dynamic values in span names, for example, GET /users/123 or query-abc-def-ghi, create a unique series for every distinct path or identifier.

The span_name_sanitization option uses the DRAIN algorithm to automatically group similar span names and replace variable segments with a placeholder. For example, GET /users/123 and GET /users/456 are both mapped to GET /users/<_>, which reduces them to a single series.

To configure span name sanitization, refer to Reduce cardinality with span name sanitization.

Manage cardinality with limits

For troubleshooting and managing cardinality limits at runtime, refer to the following sections in the metrics-generator troubleshooting documentation:

Max active series to detect and configure the active series limit
Entity-based limiting to limit by unique label combinations instead of individual series
Per-label cardinality limiting to limit cardinality (distinct values) per label

Estimate cardinality from traces

Thu, 28 May 2026 17:50:33 +0100

Estimate cardinality from traces

Cardinality can pose a problem when you have lots of services. There isn’t a direct formula or solution to this issue. The following guide should help estimate the cardinality that a feature generates.

For more information on cardinality, refer to the Cardinality documentation.

How to estimate the cardinality

The amount of edges depends on the number of nodes in the system and the direction of the requests between them. Let’s call this amount hops. Every hop is a unique combination of client + server labels.

For example:

A system with 3 nodes (A, B, C) of which A only calls B and B only calls C will have 2 hops (A → B, B → C)
A system with 3 nodes (A, B, C) that call each other (that is, all bidirectional links) will have 6 hops (A → B, B → A, B → C, C → B, A → C, C → A)

You can’t calculate the amount of hops automatically based upon the nodes, but it should be a value between #services - 1 and #services!.

If you know the amount of hops in a system, you can calculate the cardinality of the generated service graphs (assuming #hb is the number of histogram buckets):

  traces_service_graph_request_total: #hops
  traces_service_graph_request_failed_total: #hops
  traces_service_graph_request_server_seconds: #hb * #hops
  traces_service_graph_request_client_seconds: #hb * #hops
  traces_service_graph_unpaired_spans_total: #services (absolute worst case)
  traces_service_graph_dropped_spans_total: #services (absolute worst case)

Finally, you get the following cardinality estimation:

  Sum: [([2 * #hb] + 2) * #hops] + [2 * #services]

Note
If enable_messaging_system_latency_histogram configuration is set to true, another histogram is produced:
  traces_service_graph_request_messaging_system_seconds: #hb * #hops
In that case, the estimation formula would be:
  Sum: [([3 * #hb] + 2) * #hops] + [2 * #services]

Note
To estimate the number of metrics, refer to the Dry run metrics generator documentation.

Managing cardinality with limits

For troubleshooting and managing cardinality limits at runtime, refer to the following sections in the metrics-generator troubleshooting documentation:

Max active series to detect and configure the active series limit
Entity-based limiting to limit by unique label combinations instead of individual series
Per-label cardinality limiting to limit cardinality (distinct values) per label

Reduce cardinality with span name sanitization

Thu, 28 May 2026 17:50:33 +0100

Reduce cardinality with span name sanitization

Note
span name sanitization is an experimental feature. Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Enable the feature toggle in Grafana to use this feature. Do not enable this feature in production environments.

The span_name label often contributes the most to high cardinality in generated metrics. Applications that embed dynamic values in span names, for example, REST paths with user IDs (GET /users/123) or auto-generated operation names (query-abc-def-ghi), create a unique series for every distinct value. This drives up active series counts, increases storage and query costs, and can push you past cardinality limits.

You can enable span_name_sanitization to solve this without changing instrumentation across your services. The option uses the DRAIN algorithm to learn patterns from incoming span names and replace variable segments with a <_> placeholder, reducing many unique span names down to a single representative series.

Use span name sanitization when:

Your span_name label has high cardinality due to embedded IDs, timestamps, or request parameters.
You don’t control the instrumentation that produces these span names, or changing it across all services isn’t practical.
You want to reclaim active series budget without losing meaningful aggregation in your span metrics.

Before you begin

The metrics-generator must be enabled and processing spans.
You need access to the Tempo configuration or the user-configurable overrides API.

How the DRAIN span name sanitization works

DRAIN is a streaming algorithm that learns patterns from span names as they arrive. It breaks each span name into tokens (segments separated by delimiters like /, -, or spaces) and organizes them into a tree structure. When the algorithm sees enough variation at a specific position, for example, numeric IDs, UUIDs, or other data-like values, it replaces that position with a <_> wildcard.

For example, given these span names:

GET /users/123
GET /users/456
GET /users/789

DRAIN detects that the third token varies while the rest stays constant, and produces the sanitized pattern GET /users/<_>. All three original span names map to this single pattern in generated metrics.

DRAIN preserves meaningful trailing segments. For example, given GET /users/123/data and GET /users/123/settings, DRAIN sanitizes the ID but keeps /data and /settings distinct, producing GET /users/<_>/data and GET /users/<_>/settings rather than collapsing both paths into a single pattern.

The algorithm adapts continuously. As new span names arrive, DRAIN refines its patterns and merges additional variable segments. It also prunes stale patterns that haven’t been seen recently.

For more details on the algorithm, refer to Drain: An Online Log Parsing Approach with Fixed Depth Tree (He et al., IEEE ICWS 2017).

Configure the span name sanitizer

Set the span_name_sanitization option in the overrides block of your Tempo configuration:

overrides:
  defaults:
    metrics_generator:
      span_name_sanitization: "enabled"

The span_name_sanitization option accepts these values:

Value	Behavior
`""` (empty string)	Disabled. No sanitization is applied. This is the default.
`dry_run`	Trains the DRAIN model and produces the `tempo_metrics_generator_registry_post_sanitization_demand_estimate` metric, but doesn’t modify span names in generated metrics. Use this mode to evaluate the cardinality impact before enabling.
`enabled`	Applies DRAIN sanitization to span names in generated metrics.

You can also set this option per-tenant using the user-configurable overrides API.

Evaluate the impact with dry-run mode

Before enabling span_name_sanitization, use dry_run mode to understand the impact without changing your metrics:

overrides:
  defaults:
    metrics_generator:
      span_name_sanitization: "dry_run"

After the model has trained for a few minutes, compare the estimated demand against your current active series:

tempo_metrics_generator_registry_post_sanitization_demand_estimate estimates cardinality if sanitization were fully applied
tempo_metrics_generator_registry_active_series shows the current active series count

If the post-sanitization estimate is significantly lower, switching to enabled reduces your active series count.

Monitor the span name sanitizer

After you enable span_name_sanitization, use these metrics to observe its effect:

tempo_metrics_generator_registry_spans_sanitized_total counts the spans whose span_name label was replaced by a sanitized pattern
tempo_metrics_generator_registry_post_sanitization_demand_estimate shows the ongoing cardinality demand after sanitization is applied

Next steps

Cardinality for background on how cardinality impacts the metrics-generator
Estimate cardinality from traces to calculate expected active series for service graphs
Troubleshoot metrics-generator for diagnosing metrics quality issues

Multi-tenancy support

Thu, 28 May 2026 17:50:33 +0100

Multi-tenancy support

Multi-tenancy is supported in the metrics-generator through the use of environment variables and per-tenant overrides. This is useful when you want to propagate the multi-tenancy to the metrics backend, keeping the data separated and secure.

Usage

To use this feature, you need to define the remote_write_headers override for each tenant in your configuration. You can also use environment variables in your configuration file, which will be expanded at runtime. To make use of environment variables, you need to pass the --config.expand-env flag to Tempo.

Warning
Tempo 3.0 disables legacy flat (unscoped) overrides by default. If your overrides use the legacy format, either migrate to the scoped format shown below or set enable_legacy_overrides: true temporarily.

Example using the scoped overrides format:

overrides:
  defaults:
    metrics_generator:
      processors: [ 'span-metrics' ]
  team-traces-a:
    metrics_generator:
      remote_write_headers:
        Authorization: ${PROM_A_BASIC_AUTH}
  team-traces-b:
    metrics_generator:
      processors: [ 'span-metrics', 'service-graphs' ]
      remote_write_headers:
        Authorization: ${PROM_B_BEARER_AUTH}

export PROM_A_BASIC_AUTH="Basic $(echo "team-a:$(cat /token-prometheus-a)"|base64|tr -d '[:space:]')"
export PROM_B_BEARER_AUTH="Bearer $(cat /token-prometheus-b)"

In this example, PROM_A_BASIC_AUTH and PROM_B_BEARER_AUTH are environment variables that contain the respective tenants’ authorization tokens. The remote_write_headers override is used to specify the Authorization header for each tenant. The Authorization header is used to authenticate the remote write request to the Prometheus remote write endpoint.