Documentation Grafana documentation Alerting Introduction Alert rules Labels and annotations

Labels and annotations

Labels and annotations add additional information about an alert using key/value pairs:

• Labels are used to differentiate an alert from all other alerts and decide how to manage them.
• Annotations provide extra details for alert responders to help them understand and address potential issues.

Labels

Labels are unique identifiers of an alert instance. You can use them for searching, silencing, and routing notifications.

Examples of labels are server=server1 or team=backend. Each alert rule can have more than one label and the complete set of labels for an alert rule is called its label set. It is this label set that identifies the alert.

For example, an alert instance might have the label set {alertname="High CPU usage",server="server1"} while another alert instance might have the label set {alertname="High CPU usage",server="server2"}. These are two separate alert instances because although their alertname labels are the same, their server labels are different.

Labels are a fundamental component of alerting:

• The complete set of labels for an alert is what uniquely identifies an alert instance.
• The alerting UI shows labels for every alert instance generated during evaluation of that rule.
• Notification policies and silences use labels to match alert instances and route them to contact points or stop their notifications.
• Contact points can include information from labels in notification messages.

Label types

An alert’s label set can contain three types of labels:

User-configured labels

Labels that you manually configure in the alert rule to identify the generated alert instances and manage the alerts. Common custom labels, depending on the use case, are: severity, priority, team, and service.

Additionally, you can use a template to customize the label value and generate dynamic values from query data.

Data source query labels

For example, if you are monitoring temperature readings and each time series for these readings has a sensor_id, and a location label, an alert instance might have the labels {sensor_id="1",location="NY"}, while another alert instance might have {sensor_id="2",location="WA"}.

Data source query labels labels are also used to generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data.

Reserved labels

Reserved labels are automatically added by Grafana:

• alertname: the name of the alert rule.
• grafana_folder: the title of the folder containing the alert.

Labels prefixed with grafana_ are reserved by Grafana for special use. You can disable reserved labels via the unified_alerting.reserved_labels option.

Annotations

Annotations add additional information to alert instances, helping responders identify and address potential issues.

Annotations are displayed in Grafana and are included by default in notifications. Grafana provides several optional annotations that you can edit:

• summary: A short summary of what the alert has detected and why.
• description: A detailed description of what happened and what the alert does.
• runbook_url: The runbook page to guide operators managing a potential incident.
• dashboardUId and panelId: Link the alert to a dashboard and panel to facilitate alert investigation.

For example, you can edit the annotation summary to explain why the alert was triggered:

Copy

CPU usage has exceeded 80% for the last 5 minutes.

And edit the description annotation to provide more context and how to respond:

Copy

The web server's CPU has exceeded 80% for more than 5 minutes.

This indicates that the system is under heavy load and may result in an outage.

Consider scaling the server's resources and investigating bottlenecks.

Like labels, annotations can use a template to include dynamic data from queries.