Documentation Plugins Prometheus Alerting Introduction Alert rules Labels and annotations

Labels and annotations

Labels and annotations look similar in a rule definition and do completely different jobs. The short version: labels are for machines, annotations are for people.

Labels identify the alert

An alert instance’s identity is its complete set of labels. That set comes from two places: the labels on the series the expression returned, plus any labels the rule adds.

Identity matters because Alertmanager uses it for everything:

• Routing: matchers in the routing tree are label matchers. Refer to The routing tree.
• Grouping: alerts are batched by label value using group_by. Refer to Group alert notifications.
• Silencing: silences match on labels. Refer to Create a silence.
• Inhibition: source and target matchers are label matchers, and equal compares label values. Refer to Configure inhibition rules.
• Deduplication: two alerts with identical labels are the same alert.

Because identity is the whole label set, changing any label produces a different alert. An alert whose labels change mid-incident looks to Alertmanager like the old one resolving and a new one appearing, which can re-notify people. Avoid putting anything that varies over time—a timestamp, a current value—in a label.

Conventional labels

alertname is set automatically from the rule’s name and is what most routing keys off first. Beyond that, the useful convention is a small, stable set that every rule sets consistently:

YAML Copy

labels:
  severity: critical
  team: platform

The value is in the consistency. A routing tree can only send critical alerts to PagerDuty if every rule agrees on what severity: critical means and spells it the same way. Refer to Best practices.

Labels and cardinality

Every distinct label combination is a distinct alert. A label with many possible values—a request ID, a user ID, a pod name in a large cluster—multiplies the number of alerts a single rule can produce. Aggregate those away in the expression rather than carrying them into the alert.

Annotations describe the alert

Annotations carry the human-readable content: what happened, what it means, what to do about it. They’re passed through to notifications and are what people actually read.

YAML Copy

annotations:
  summary: 'High latency on {{ $labels.job }}'
  description: '{{ $labels.job }} has a mean latency of {{ $value | printf "%.2f" }}s over 5m.'
  runbook_url: https://runbooks.example.com/high-latency

Annotations play no part in routing, grouping, silencing, or deduplication. Changing an annotation on a firing alert doesn’t create a new alert, which is exactly why anything that varies belongs here rather than in a label.

Annotation values are templated by the ruler at evaluation time, so $labels and $value are available. Refer to Template annotations and labels.

Which one should it be?

Ask what the field is for:

In the plugin, both are edited in the rule editor: labels in the rule definition, annotations under Annotations, where Add custom annotation adds keys beyond the standard ones.