Correlate signals on Grafana Labs

Put detail on the right signal

Thu, 06 Aug 2026 10:05:27 -0400

Put detail on the right signal

Sometimes you genuinely need a high-cardinality value, like a request ID, when you’re debugging. The answer isn’t to delete it, and it isn’t to put it on a metric label. It’s to put the value on the signal designed to hold it, and correlate across signals when you investigate.

The pattern

When a high-cardinality value is attached to a metric label, every unique value multiplies your active series. The same value attached to a log costs a fraction of that, because Loki doesn’t index log content. Here’s how to use structured metadata to keep your metrics low-cardinality and by leveraging logs:

Keep the metric low-cardinality. The metric answers “how many” and “how fast” per bounded dimension, such as service and status.
Put the high-cardinality detail on the correlated log. Attach it as structured metadata, or leave it in the log line. The log answers “which exact request”.
Correlate at query time. Shared bounded labels, such as service and env, connect the metric that alerted you to the logs that explain it.

A working example

Thu, 06 Aug 2026 10:05:27 -0400

A working example

Suppose you track request latency and want to find slow requests for specific users.

Instead of http_request_duration_seconds{user_id="u-48221", ...}, which creates a series per user per label combination,
keep the metric as http_request_duration_seconds{service="checkout", env="prod"},
and log each request with user_id and request_id attached as structured metadata.

When latency spikes, the metric tells you where and when. Then you query the logs for that service and window, filtering by structured metadata: {app="checkout"} | user_id="u-48221". You keep full debugging detail without paying for it on every series.

Take it further with traces

Thu, 06 Aug 2026 10:05:27 -0400

Take it further with traces

If you also collect traces, the same pattern extends to a third signal using exemplars. An exemplar is a trace ID attached alongside a metric sample, not as a label, so the metric stays low-cardinality while a latency spike becomes one click away from the exact slow trace.

Attach the trace ID to the metric as an exemplar, not a label.
Keep the trace ID on logs as structured metadata, so logs link to the same trace.

Exemplars require a tracing backend such as Tempo, and exemplar support enabled on your metrics data source.

Why this works

Thu, 06 Aug 2026 10:05:27 -0400

Why this works

Each signal is priced and indexed for a different job. Metrics are cheap per series but multiply by label combinations. Logs carry arbitrary detail, and structured metadata makes high-cardinality fields queryable without indexing them. Traces hold per-request detail natively, and exemplars link them to your metrics without adding labels. Putting each value where it belongs is what a labeling strategy actually is; not fewer data points, just the right home for each one.

What you’ve done

Throughout this journey, you have:

Learned how label combinations multiply into unique active series and log streams, and that this is what you’re billed for
Explored your own stack’s cardinality with the Cardinality tab in the Cost Management and Billing app
Learned the rules for metric labels: bounded values only, added only when queried, and audited at ingestion setup
Reviewed Adaptive Metrics recommendations to reduce cardinality you already have
Applied the logs decision framework: labels for origin, structured metadata for high-cardinality fields, filter expressions for the rest
Learned to correlate signals so high-cardinality detail lives on logs and traces, not metric labels

Two questions to keep asking

Keep the two questions from this journey in your team’s review habits: Is this label value bounded? and Will anyone query by it? Most cardinality problems never happen when someone asks those questions at instrumentation time.

Resources

Thu, 06 Aug 2026 10:05:27 -0400

Correlate signals on Grafana Labs

Put detail on the right signal

Put detail on the right signal

The pattern

A working example

A working example

Take it further with traces

Take it further with traces

Why this works

Why this works

What you’ve done

Two questions to keep asking

Resources

Documentation

Community