Explore Alerting on Grafana Labs

Data sources

Tue, 10 Mar 2026 13:40:23 +0000

Data sources

There are a number of data sources that are compatible with Grafana Alerting. Each data source is supported by a plugin. You can use one of the built-in data sources listed below, use external data source plugins, or create your own data source plugin.

If you are creating your own data source plugin, make sure it is a backend plugin as Grafana Alerting requires this in order to be able to evaluate rules using the data source. Frontend data sources are not supported, because the evaluation engine runs on the backend.

Specifying { “alerting”: true, “backend”: true } in the plugin.json file indicates that the data source plugin is compatible with Grafana Alerting and includes the backend data-fetching code. For more information, refer to Build a data source backend plugin.

These are the data sources that are compatible with and supported by Grafana Alerting.

Useful links

Alert rules

Tue, 10 Mar 2026 13:40:23 +0000

About alert rules

An alerting rule is a set of evaluation criteria that determines whether an alert instance will fire. The rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and optionally, the duration over which the condition is met.

While queries and expressions select the data set to evaluate, a condition sets the threshold that an alert must meet or exceed to create an alert.

An interval specifies how frequently an alerting rule is evaluated. Duration, when configured, indicates how long a condition must be met. The alert rules can also define alerting behavior in the absence of data.

Alerting on numeric data

Tue, 10 Mar 2026 13:40:23 +0000

Alerting on numeric data

This topic describes how Grafana managed alerts are evaluated by the backend engine as well as how Grafana handles alerting on numeric rather than time series data.

Alerting on numeric data
- Alert evaluation
  - Metrics from the alerting engine
- Alerting on numeric data
  - Tabular Data
  - Example

Alert evaluation

Grafana managed alerts query the following backend data sources that have alerting enabled:

built-in data sources or those developed and maintained by Grafana: Graphite, Prometheus, Loki, InfluxDB, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle, and Azure Monitor
community developed backend data sources with alerting enabled (backend and alerting properties are set in the plugin.json

Metrics from the alerting engine

The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes internal metrics.

Metric Name	Type	Description
`grafana_alerting_alerts`	gauge	How many alerts by state
`grafana_alerting_request_duration`	histogram	Histogram of requests to the Alerting API
`grafana_alerting_active_configurations`	gauge	The number of active, non default Alertmanager configurations for grafana managed alerts
`grafana_alerting_rule_evaluations_total`	counter	The total number of rule evaluations
`grafana_alerting_rule_evaluation_failures_total`	counter	The total number of rule evaluation failures
`grafana_alerting_rule_evaluation_duration`	summary	The duration for a rule to execute
`grafana_alerting_rule_group_rules`	gauge	The number of rules

Alerting on numeric data

Among certain data sources numeric data that is not time series can be directly alerted on, or passed into Server Side Expressions (SSE). This allows for more processing and resulting efficiency within the data source, and it can also simplify alert rules. When alerting on numeric data instead of time series data, there is no need to reduce each labeled time series into a single number. Instead labeled numbers are returned to Grafana instead.

Tabular Data

This feature is supported with backend data sources that query tabular data:

SQL data sources such as MySQL, Postgres, MSSQL, and Oracle.
The Azure Kusto based services: Azure Monitor (Logs), Azure Monitor (Azure Resource Graph), and Azure Data Explorer.

A query with Grafana managed alerts or SSE is considered numeric with these data sources, if:

The “Format AS” option is set to “Table” in the data source query.
The table response returned to Grafana from the query includes only one numeric (e.g. int, double, float) column, and optionally additional string columns.

If there are string columns then those columns become labels. The name of column becomes the label name, and the value for each row becomes the value of the corresponding label. If multiple rows are returned, then each row should be uniquely identified their labels.

Example

For a MySQL table called “DiskSpace”:

Time	Host	Disk	PercentFree
2021-June-7	web1	/etc	3
2021-June-7	web2	/var	4
2021-June-7	web3	/var	8
…	…	…	…

You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:

SELECT Host, Disk, CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
  SELECT
      Host,
      Disk,
      Avg(PercentFree)
  FROM DiskSpace
  Group By
    Host,
    Disk
  Where __timeFilter(Time)

This query returns the following Table response to Grafana:

Host	Disk	PercentFree
web1	/etc	3
web2	/var	4
web3	/var	0

When this query is used as the condition in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:

Labels	Status
{Host=web1,disk=/etc}	Alerting
{Host=web2,disk=/var}	Alerting
{Host=web3,disk=/var}	Normal

Labels and annotations

Tue, 10 Mar 2026 13:40:23 +0000

Labels and annotations

Labels and annotations contain information about an alert. Both labels and annotations have the same structure: a set of named values; however their intended uses are different. An example of label, or the equivalent annotation, might be alertname="test".

The main difference between a label and an annotation is that labels are used to differentiate an alert from all other alerts, while annotations are used to add additional information to an existing alert.

For example, consider two high CPU alerts: one for server1 and another for server2. In such an example we might have a label called server where the first alert has the label server="server1" and the second alert has the label server="server2". However, we might also want to add a description to each alert such as "The CPU usage for server1 is above 75%.", where server1 and 75% are replaced with the name and CPU usage of the server (please refer to the documentation on templating labels and annotations for how to do this). This kind of description would be more suitable as an annotation.

Labels

Labels contain information that identifies an alert. An example of a label might be server=server1. Each alert can have more than one label, and the complete set of labels for an alert is called its label set. It is this label set that identifies the alert.

For example, an alert might have the label set {alertname="High CPU usage",server="server1"} while another alert might have the label set {alertname="High CPU usage",server="server2"}. These are two separate alerts because although their alertname labels are the same, their server labels are different.

The label set for an alert is a combination of the labels from the datasource, custom labels from the alert rule, and a number of reserved labels such as alertname.

Custom Labels

Custom labels are additional labels from the alert rule. Like annotations, custom labels must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. Documentation on how to template custom labels can be found here.

When using custom labels with templates it is important to make sure that the label value does not change between consecutive evaluations of the alert rule as this will end up creating large numbers of distinct alerts. However, it is OK for the template to produce different label values for different alerts. For example, do not put the value of the query in a custom label as this will end up creating a new set of alerts each time the value changes. Instead use annotations.

It is also important to make sure that the label set for an alert does not have two or more labels with the same name. If a custom label has the same name as a label from the datasource then it will replace that label. However, should a custom label have the same name as a reserved label then the custom label will be omitted from the alert.

Annotations

Annotations are named pairs that add additional information to existing alerts. There are a number of suggested annotations in Grafana such as description, summary, runbook_url, dashboardUId and panelId. Like custom labels, annotations must have a name, and their value can contain a combination of text and template code that is evaluated when an alert is fired. If an annotation contains template code, the template is evaluated once when the alert is fired. It is not re-evaluated, even when the alert is resolved. Documentation on how to template annotations can be found here.

State and health of alerting rules

Tue, 10 Mar 2026 13:40:23 +0000

State and health of alerting rules

The state and health of alerting rules help you understand several key status indicators about your alerts.

There are three key components: alert rule state, alert instance state, and alert rule health. Although related, each component conveys subtly different information.

Alert rule state

An alert rule can be in either of the following states:

State	Description
Normal	None of the time series returned by the evaluation engine is in a `Pending` or `Firing` state.
Pending	At least one time series returned by the evaluation engine is `Pending`.
Firing	At least one time series returned by the evaluation engine is `Firing`.

Note: Alerts will transition first to pending and then firing, thus it will take at least two evaluation cycles before an alert is fired.

Alert instance state

An alert instance can be in either of the following states:

State	Description
Normal	The state of an alert that is neither firing nor pending, everything is working correctly.
Pending	The state of an alert that has been active for less than the configured threshold duration.
Alerting	The state of an alert that has been active for longer than the configured threshold duration.
NoData	No data has been received for the configured time window.
Error	The error that occurred when attempting to evaluate an alerting rule.

Alert rule health

An alert rule can have one the following health statuses:

State	Description
Ok	No error when evaluating an alerting rule.
Error	An error occurred when evaluating an alerting rule.
NoData	The absence of data in at least one time series returned during a rule evaluation.

Special alerts for `NoData` and `Error`

When evaluation of an alerting rule produces state NoData or Error, Grafana Alerting will generate alert instances that have the following additional labels:

Label	Description
alertname	Either `DatasourceNoData` or `DatasourceError` depending on the state.
datasource_uid	The UID of the data source that caused the state.

You can handle these alerts the same way as regular alerts by adding a silence, route to a contact point, and so on.

Contact points

Tue, 10 Mar 2026 13:40:23 +0000

Contact points

Use contact points to define how your contacts are notified when an alert rule fires. A contact point can have one or more contact point types, for example, email, slack, webhook, and so on. When an alert rule fires, a notification is sent to all contact point types listed for a contact point. Contact points can be configured for the Grafana Alertmanager as well as external alertmanagers.

You can also use notification templating to customize notification messages for contact point types.

Note:

If you’ve created an OnCall contact point in the Grafana OnCall application, you can view it in the Alerting application.

Supported contact point integrations

The following table lists the contact point integrations supported by Grafana.

Name	Type	Grafana Alertmanager	Other Alertmanagers
DingDing	`dingding`	Supported	N/A
Discord	`discord`	Supported	N/A
Email	`email`	Supported	Supported
Google Hangouts	`googlechat`	Supported	N/A
Kafka	`kafka`	Supported	N/A
Line	`line`	Supported	N/A
Microsoft Teams	`teams`	Supported	N/A
Opsgenie	`opsgenie`	Supported	Supported
Pagerduty	`pagerduty`	Supported	Supported
Prometheus Alertmanager	`prometheus-alertmanager`	Supported	N/A
Pushover	`pushover`	Supported	Supported
Sensu Go	`sensugo`	Supported	N/A
Slack	`slack`	Supported	Supported
Telegram	`telegram`	Supported	N/A
Threema	`threema`	Supported	N/A
VictorOps	`victorops`	Supported	Supported
Webhook	`webhook`	Supported	Supported (different format)
Cisco Webex Teams	`webex`	Supported	Supported
WeCom	`wecom`	Supported	N/A
Zenduty	`webhook`	Supported	N/A

Useful links

Manage contact points

Create and edit notification templates

Notifications

Tue, 10 Mar 2026 13:40:23 +0000

Notifications

Grafana uses Alertmanagers to send notifications for firing and resolved alerts. Grafana has its own Alertmanager, referred to as “Grafana” in the user interface, but also supports sending notifications from other Alertmanagers too, such as the Prometheus Alertmanager. The Grafana Alertmanager uses notification policies and contact points to configure how and where a notification is sent; how often a notification should be sent; and whether alerts should all be sent in the same notification, sent in grouped notifications based on a set of labels, or as separate notifications.

Notification policies

Notification policies control when and where notifications are sent. A notification policy can choose to send all alerts together in the same notification, send alerts in grouped notifications based on a set of labels, or send alerts as separate notifications. You can configure each notification policy to control how often notifications should be sent as well as having one or more mute timings to inhibit notifications at certain times of the day and on certain days of the week.

Notification policies are organized in a tree structure where at the root of the tree there is a notification policy called the root policy. There can be only one root policy and the root policy cannot be deleted.

Specific routing policies are children of the root policy and can be used to match either all alerts or a subset of alerts based on a set of matching labels. A notification policy matches an alert when its matching labels match the labels in the alert.

A specific routing policy can have its own child policies, called nested policies, which allow for additional matching of alerts. An example of a specific routing policy could be sending infrastructure alerts to the Ops team; while a child policy might send high priority alerts to Pagerduty and low priority alerts as emails.

All alerts, irrespective of their labels, match the root policy. However, when the root policy receives an alert it looks at each specific routing policy and sends the alert to the first specific routing policy that matches the alert. If the specific routing policy has further child policies, then it can attempt to the match the alert against one of its nested policies. If no nested policies match the alert then the specific routing policy is the matching policy. If there are no specific routing policies, or no specific routing policies match the alert, then the root policy is the matching policy.

Contact points

Contact points contain the configuration for sending notifications. A contact point is a list of integrations, each of which sends a notification to a particular email address, service or URL. Contact points can have multiple integrations of the same kind, or a combination of integrations of different kinds. For example, a contact point could contain a Pagerduty integration; an email and Slack integration; or a Pagerduty integration, a Slack integration, and two email integrations. You can also configure a contact point with no integrations; in which case no notifications are sent.

A contact point cannot send notifications until it has been added to a notification policy. A notification policy can only send alerts to one contact point, but a contact point can be added to a number of notification policies at the same time. When an alert matches a notification policy, the alert is sent to the contact point in that notification policy, which then sends a notification to each integration in its configuration.

Supported integrations

The following table contains the integrations supported in Grafana:

Name	Type	Grafana Alertmanager	Other Alertmanagers
DingDing	`dingding`	Supported	N/A
Discord	`discord`	Supported	N/A
Email	`email`	Supported	Supported
Google Hangouts	`googlechat`	Supported	N/A
Kafka	`kafka`	Supported	N/A
Line	`line`	Supported	N/A
Microsoft Teams	`teams`	Supported	N/A
Opsgenie	`opsgenie`	Supported	Supported
Pagerduty	`pagerduty`	Supported	Supported
Prometheus Alertmanager	`prometheus-alertmanager`	Supported	N/A
Pushover	`pushover`	Supported	Supported
Sensu Go	`sensugo`	Supported	N/A
Slack	`slack`	Supported	Supported
Telegram	`telegram`	Supported	N/A
Threema	`threema`	Supported	N/A
VictorOps	`victorops`	Supported	Supported
Webhook	`webhook`	Supported	Supported (different format)
Cisco Webex Teams	`webex`	Supported	Supported
WeCom	`wecom`	Supported	N/A
Zenduty	`webhook`	Supported	N/A

Templating notifications

You can customize notifications with templates. For example, templates can be used to change the subject and message of an email, or the title and message of notifications sent to Slack.

Templates are not limited to an individual integration or contact point, but instead can be used in a number of integrations in the same contact point and even integrations across different contact points. For example, a Grafana user can create a template called custom_subject_or_title and use it for both templating subjects in emails and titles of Slack messages without having to create two separate templates.

All notifications templates are written in Go’s templating language, and are in the Contact points tab on the Alerting page.

Silences

You can use silences to mute notifications from one or more firing rules. Silences do not stop alerts from firing or being resolved, or hide firing alerts in the user interface. A silence lasts as long as its duration which can be configured in minutes, hours, days, months or years.

Alerting high availability

Tue, 10 Mar 2026 13:40:23 +0000

Alerting high availability

The Grafana Alerting system has two main components: a Scheduler and an internal Alertmanager. The Scheduler evaluates your alert rules, while the internal Alertmanager manages routing and grouping.

When running Grafana Alerting in high availability, the operational mode of the scheduler remains unaffected, and each Grafana instance evaluates all alerts. The operational change happens in the Alertmanager when it deduplicates alert notifications across Grafana instances.

The coordination between Grafana instances happens via a Gossip protocol. Alerts are not gossiped between instances and each scheduler delivers the same volume of alerts to each Alertmanager.

The two types of messages gossiped between Grafana instances are:

Notification logs: Who (which instance) notified what (which alert).
Silences: If an alert should fire or not.

The notification logs and silences are persisted in the database periodically and during a graceful Grafana shut down.

Useful links

Configure alerting high availability

Explore Alerting on Grafana Labs

Data sources

Data sources

Useful links

Alert rules

About alert rules

Alerting on numeric data

Alerting on numeric data

Alert evaluation

Metrics from the alerting engine

Alerting on numeric data

Tabular Data

Example

Labels and annotations

Labels and annotations

Labels

Custom Labels

Annotations

State and health of alerting rules

State and health of alerting rules

Alert rule state

Alert instance state

Alert rule health

Special alerts for NoData and Error

Contact points

Contact points

Supported contact point integrations

Useful links

Notifications

Notifications

Notification policies

Contact points

Supported integrations

Templating notifications

Silences

Alerting high availability

Alerting high availability

Useful links

Special alerts for `NoData` and `Error`