Alerts on Grafana Labs

What's New with Grafana 8 Alerts

Thu, 05 Aug 2021 18:36:26 +0200

What’s New with Grafana 8 Alerts

The Alerts released with Grafana 8.0 are an opt-in feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible datasources in one UI and API. You are able to create and edit alerting rules for Grafana managed alerts, Cortex alerts, and Loki alerts as well as see alerting information from prometheus-compatible datasources in a single, searchable view.

Multi-dimensional alerting

Create alerts that will give you system-wide visibility with a single alerting rule. With Grafana 8 alerts, you are able to generate multiple alert instances from a single rule eg. creating a rule to monitor disk usage for multiple mount points on a single host. The evaluation engine is able to return multiple time series from a single query. Each time series is identified by its label set.

Create alerts outside of Dashboards

Grafana legacy alerts were tied to a dashboard. Grafana 8 Alerts allow you to create queries and expressions that can combine data from multiple sources, in unique ways. You are still able to link dashboards and panels to alerting rules, allowing you to quickly troubleshoot the system under observation, by linking a dashboard and/or panel ID to the alerting rule.

Create Loki and Cortex alerting rules

With Grafana 8 Alerts you are able to manage your Loki and Cortex alerting rules using the same UI and API as your Grafana managed alerts.

View and search for alerts from Prometheus

You can now display all of your alerting information in one, searchable UI. Alerts for Prometheus compatible datasources are listed below Grafana managed alerts. Search for labels across multiple datasources to quickly find all of the relevant alerts.

Grafana 8 Alerts

Fri, 07 Mar 2025 11:34:44 +0100

Overview of Grafana 8 alerts

Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.

Note: This information is for the new, Grafana 8 Alerts. This is an opt-in feature released in Grafana 8.0. Grafana still supports legacy dashboard alerts out of the box

Alerts have four main components:

Alerting rule - One or more query and/or expression, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert.
Contact point - A channel for sending notifications when the conditions of an alerting rule are met.
Notification policy - A set of matching and grouping criteria used to determine where, and how frequently, to send notifications.
Silences - Date and matching criteria used to silence notifications.

Alerting tasks

You can perform the following tasks for alerts:

Clustering

The current alerting system doesn’t support high availability. Alert notifications are not deduplicated and load balancing is not supported between instances e.g. silences from one instance will not appear in the other. The Grafana team aims to have this feature by Grafana version 8.1+.

Alert evaluation

Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.

Alerting rules can only query backend data sources with alerting enabled:

builtin or developed and maintained by grafana: Graphite, Prometheus, Loki, InfluxDB, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle, and Azure Data Explorer
any community backend data sources with alerting enabled (backend and alerting properties are set in the plugin.json)

Metrics from the alerting engine

The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes internal metrics.

Metric Name	Type	Description
`alerting.alerts`	gauge	How many alerts by state
`alerting.request_duration_seconds`	histogram	Histogram of requests to the Alerting API
`alerting.active_configurations`	gauge	The number of active, non default Alertmanager configurations for grafana managed alerts
`alerting.rule_evaluations_total`	counter	The total number of rule evaluations
`alerting.rule_evaluation_failures_total`	counter	The total number of rule evaluation failures
`alerting.rule_evaluation_duration_seconds`	summary	The duration for a rule to execute
`alerting.rule_group_rules`	gauge	The number of rules

View alert rules and their current state

Legacy Grafana Alerts

Fri, 07 Mar 2025 11:34:44 +0100

Legacy Grafana alerts

Grafana 8.0 has new and improved alerts. The new alerting system are an opt-in feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible data sources in one UI and API.

Out of the box, Grafana still supports legacy dashboard alerts. Legacy Grafana alerts consists of two parts:

Alert rules - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana. Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.

Currently only the graph panel visualization supports alerts.

Legacy alerts have two main components:

Alert rule - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana.
Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.

Alert tasks

You can perform the following tasks for alerts:

Clustering

Currently alerting supports a limited form of high availability. Since v4.2.0 of Grafana, alert notifications are deduped when running multiple servers. This means all alerts are executed on every server but no duplicate alert notifications are sent due to the deduping logic. Proper load balancing of alerts will be introduced in the future.

Alert evaluation

Alert rules can only query backend data sources with alerting enabled:

builtin or developed and maintained by grafana: Graphite, Prometheus, Loki, InfluxDB, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle, and Azure Data Explorer
any community backend data sources with alerting enabled (backend and alerting properties are set in the plugin.json)

Metrics from the alert engine

The alert engine publishes some internal metrics about itself. You can read more about how Grafana publishes internal metrics.

Metric Name	Type	Description
`alerting.alerts`	gauge	How many alerts by state
`alerting.request_duration_seconds`	histogram	Histogram of requests to the Alerting API
`alerting.active_configurations`	gauge	The number of active, non default alertmanager configurations for grafana managed alerts
`alerting.rule_evaluations_total`	counter	The total number of rule evaluations
`alerting.rule_evaluation_failures_total`	counter	The total number of rule evaluation failures
`alerting.rule_evaluation_duration_seconds`	summary	The duration for a rule to execute
`alerting.rule_group_rules`	gauge	The number of rules