Slide 1 of 5

Put detail on the right signal

Put detail on the right signal

Sometimes you genuinely need a high-cardinality value, like a request ID, when you’re debugging. The answer isn’t to delete it, and it isn’t to put it on a metric label. It’s to put the value on the signal designed to hold it, and correlate across signals when you investigate.

The pattern

When a high-cardinality value is attached to a metric label, every unique value multiplies your active series. The same value attached to a log costs a fraction of that, because Loki doesn’t index log content. Here’s how to use structured metadata to keep your metrics low-cardinality and by leveraging logs:

  • Keep the metric low-cardinality. The metric answers “how many” and “how fast” per bounded dimension, such as service and status.
  • Put the high-cardinality detail on the correlated log. Attach it as structured metadata, or leave it in the log line. The log answers “which exact request”.
  • Correlate at query time. Shared bounded labels, such as service and env, connect the metric that alerted you to the logs that explain it.

Script

Sometimes you genuinely need a high-cardinality value. When you’re debugging, a request ID is exactly what you want. The answer isn’t to delete it, and it definitely isn’t to put it on a metric label.

Attached to a metric label, every unique value multiplies your active series. Attached to a log, the same value costs a fraction of that, because Loki doesn’t index log content.

Keep the metric low-cardinality, answering how many and how fast per bounded dimension, like service and status.

Put the high-cardinality detail on the correlated log, as structured metadata or in the log line, where it answers which exact request.

Correlate at query time. Shared bounded labels like service and environment connect the metric that alerted you to the logs that explain it.

Next