Slide 2 of 5

A working example

A working example

Suppose you track request latency and want to find slow requests for specific users.

  • Instead of http_request_duration_seconds{user_id="u-48221", ...}, which creates a series per user per label combination,
  • keep the metric as http_request_duration_seconds{service="checkout", env="prod"},
  • and log each request with user_id and request_id attached as structured metadata.

When latency spikes, the metric tells you where and when. Then you query the logs for that service and window, filtering by structured metadata: {app="checkout"} | user_id="u-48221". You keep full debugging detail without paying for it on every series.

Script

Here’s the pattern applied to something concrete. You track request latency, and you want to find slow requests for specific users.

The tempting move is a user ID label on the latency metric. That creates a series per user, per label combination.

Instead, keep the metric to bounded dimensions like service and environment, and log each request with the user ID and request ID attached as structured metadata.

When latency spikes, the metric tells you where and when. Then you query the logs for that service and time window, filtering by structured metadata to reach the exact user. You keep full debugging detail without paying for it on every series.

Previous Next