Alerting on Grafana Labs

Alerting Engine and Rules Guide

Fri, 06 Mar 2026 22:21:21 +0000

Alerting Engine and Rules Guide

Alerting is only available in Grafana v4.0 and above.

Introduction

Alerting in Grafana allows you to attach rules to your dashboard panels. When you save the dashboard Grafana will extract the alert rules into a separate alert rule storage and schedule them for evaluation.

In the alert tab of the graph panel you can configure how often the alert rule should be evaluated and the conditions that need to be met for the alert to change state and trigger its notifications.

Execution

The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part of core Grafana. Only some data sources are supported right now. They include Graphite, Prometheus, InfluxDB, Elasticsearch, Stackdriver, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle and Azure Data Explorer.

Alerting support for Azure Monitor is only available in Grafana v6.0 and above.

Clustering

Currently alerting supports a limited form of high availability. Since v4.2.0 of Grafana, alert notifications are deduped when running multiple servers. This means all alerts are executed on every server but no duplicate alert notifications are sent due to the deduping logic. Proper load balancing of alerts will be introduced in the future.

Rule Config

Currently only the graph panel supports alert rules but this will be added to the Singlestat and Table panels as well in a future release.

Name and Evaluation interval

Here you can specify the name of the alert rule and how often the scheduler should evaluate the alert rule.

For

Important note regarding No Data:

Do not use For with the If no data or all values are null setting set to No Data. The triggering of No Data will trigger instantly and not take For into consideration. This may also result in that an OK notification not being sent if alert transitions from No Data -> Pending -> OK.

If an alert rule has a configured For and the query violates the configured threshold it will first go from OK to Pending. Going from OK to Pending Grafana will not send any notifications. Once the alert rule has been firing for more than For duration, it will change to Alerting and send alert notifications.

Typically, it’s always a good idea to use this setting since it’s often worse to get false positive than wait a few minutes before the alert notification triggers. Looking at the Alert list or Alert list panels you will be able to see alerts in pending state.

Below you can see an example timeline of an alert using the For setting. At ~16:04 the alert state changes to Pending and after 4 minutes it changes to Alerting which is when alert notifications are sent. Once the series falls back to normal the alert rule goes back to OK.

Conditions

Currently the only condition type that exists is a Query condition that allows you to specify a query letter, time range and an aggregation function.

Query condition example

avg() OF query(A, 15m, now) IS BELOW 14

avg() Controls how the values for each series should be reduced to a value that can be compared against the threshold. Click on the function to change it to another aggregation function.
query(A, 15m, now) The letter defines what query to execute from the Metrics tab. The second two parameters define the time range, 15m, now means 15 minutes ago to now. You can also do 10m, now-2m to define a time range that will be 10 minutes ago to 2 minutes ago. This is useful if you want to ignore the last 2 minutes of data.
IS BELOW 14 Defines the type of threshold and the threshold value. You can click on IS BELOW to change the type of threshold.

The query used in an alert rule cannot contain any template variables. Currently we only support AND and OR operators between conditions and they are executed serially. For example, we have 3 conditions in the following order: condition:A(evaluates to: TRUE) OR condition:B(evaluates to: FALSE) AND condition:C(evaluates to: TRUE) so the result will be calculated as ((TRUE OR FALSE) AND TRUE) = TRUE.

We plan to add other condition types in the future, like Other Alert, where you can include the state of another alert in your conditions, and Time Of Day.

Multiple Series

If a query returns multiple series then the aggregation function and threshold check will be evaluated for each series. What Grafana does not do currently is track alert rule state per series. This has implications that are detailed in the scenario below.

Alert condition with query that returns 2 series: server1 and server2
server1 series cause the alert rule to fire and switch to state Alerting
Notifications are sent out with message: load peaking (server1)
In a subsequence evaluation of the same alert rule the server2 series also cause the alert rule to fire
No new notifications are sent as the alert rule is already in state Alerting.

So as you can see from the above scenario Grafana will not send out notifications when other series cause the alert to fire if the rule already is in state Alerting. To improve support for queries that return multiple series we plan to track state per series in a future release.

Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will be included in the reminder notification. Depending on what notification channel you’re using you may be able to take advantage of this feature for identifying new/existing series causing alert to fire. Read more about notification reminders here.

No Data / Null values

Below your conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.

No Data Option	Description
NoData	Set alert rule state to `NoData`
Alerting	Set alert rule state to `Alerting`
Keep Last State	Keep the current alert rule state, what ever it is.

Execution errors or timeouts

The last option tells how to handle execution or timeout errors.

Error or timeout option	Description
Alerting	Set alert rule state to `Alerting`
Keep Last State	Keep the current alert rule state, what ever it is.

If you have an unreliable time series store from which queries sometime timeout or fail randomly you can set this option to Keep Last State in order to basically ignore them.

Notifications

In alert tab you can also specify alert rule notifications along with a detailed message about the alert rule. The message can contain anything, information about how you might solve the issue, link to runbook, etc.

The actual notifications are configured and shared between multiple alerts. Read the notifications guide for how to configure and setup notifications.

Alert State History and Annotations

Alert state changes are recorded in the internal annotation table in Grafana’s database. The state changes are visualized as annotations in the alert rule’s graph panel. You can also go into the State history submenu in the alert tab to view and clear state history.

Troubleshooting

First level of troubleshooting you can do is hit the Test Rule button. You will get result back that you can expand to the point where you can see the raw data that was returned from your query.

Further troubleshooting can also be done by inspecting the grafana-server log. If it’s not an error or for some reason the log does not say anything you can enable debug logging for some relevant components. This is done in Grafana’s ini config file.

Example showing loggers that could be relevant when troubleshooting alerting.

[log]
filters = alerting.scheduler:debug \
          alerting.engine:debug \
          alerting.resultHandler:debug \
          alerting.evalHandler:debug \
          alerting.evalContext:debug \
          alerting.extractor:debug \
          alerting.notifier:debug \
          alerting.notifier.slack:debug \
          alerting.notifier.pagerduty:debug \
          alerting.notifier.email:debug \
          alerting.notifier.webhook:debug \
          tsdb.graphite:debug \
          tsdb.prometheus:debug \
          tsdb.opentsdb:debug \
          tsdb.influxdb:debug \
          tsdb.elasticsearch:debug \
          tsdb.elasticsearch.client:debug \

If you want to log raw query sent to your TSDB and raw response in log you also have to set grafana.ini option app_mode to development.

Alerting Metrics

Fri, 06 Mar 2026 22:21:21 +0000

Metrics from the alert engine

Alerting is only available in Grafana v4.0 and above.

The alert engine publishes some internal metrics about itself. You can read more about how Grafana publishes internal metrics.

Description	Type	Metric name
Total number of alerts	counter	`alerting.active_alerts`
Alert execution result	counter	`alerting.result`
Notifications sent counter	counter	`alerting.notifications_sent`
Alert execution timer	timer	`alerting.execution_time`

Alerting Notifications

Fri, 06 Mar 2026 22:21:21 +0000

Alert Notifications

Alerting is only available in Grafana v4.0 and above.

When an alert changes state, it sends out notifications. Each alert rule can have multiple notifications. In order to add a notification to an alert rule you first need to add and configure a notification channel (can be email, PagerDuty or other integration). This is done from the Notification Channels page.

Notification Channel Setup

On the Notification Channels page hit the New Channel button to go the page where you can configure and setup a new Notification Channel.

You specify a name and a type, and type specific options. You can also test the notification to make sure it’s setup correctly.

Default (send on all alerts)

When checked, this option will notify for all alert rules - existing and new.

Send reminders

Only available in Grafana v5.3 and above.

When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders should be sent using number of seconds (s), minutes (m) or hours (h), for example 30s, 3m, 5m or 1h etc.

Important: Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.

These examples show how often and when reminders are sent for a triggered alert.

Alert rule evaluation interval	Send reminders every	Reminder sent every (after last alert notification)
`30s`	`15s`	~30 seconds
`1m`	`5m`	~5 minutes
`5m`	`15m`	~15 minutes
`6m`	`20m`	~24 minutes
`1h`	`15m`	~1 hour
`1h`	`2h`	~2 hours

Disable resolve message

When checked, this option will disable resolve message [OK] that is sent when alerting state returns to false.

Supported Notification Types

Grafana ships with the following set of notification types:

Email

To enable email notifications you have to setup SMTP settings in the Grafana config. Email notifications will upload an image of the alert graph to an external image destination if available or fallback to attaching the image to the email. Be aware that if you use the local image storage email servers and clients might not be able to access the image.

Slack

To set up slack you need to configure an incoming webhook url at slack. You can follow their guide on how to do that here. If you want to include screenshots of the firing alerts in the Slack messages you have to configure either the external image destination in Grafana, or a bot integration via Slack Apps. Follow Slack’s guide to set up a bot integration and use the token provided (https://api.slack.com/bot-users), which starts with “xoxb”.

Setting	Description
Url	Slack incoming webhook url.
Username	Set the username for the bot’s message.
Recipient	Allows you to override the Slack recipient.
Icon emoji	Provide an emoji to use as the icon for the bot’s message. Ex :smile:
Icon URL	Provide a url to an image to use as the icon for the bot’s message.
Mention	make it possible to include a mention in the Slack notification sent by Grafana. Ex @here or @channel
Token	If provided, Grafana will upload the generated image via Slack’s file.upload API method, not the external image destination.

If you are using the token for a slack bot, then you have to invite the bot to the channel you want to send notifications and add the channel to the recipient field.

PagerDuty

To set up PagerDuty, all you have to do is to provide an API key.

Setting	Description
Integration Key	Integration key for PagerDuty.
Auto resolve incidents	Resolve incidents in PagerDuty once the alert goes back to ok

Webhook

The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint. Using this notification you could integrate Grafana into a system of your choosing.

Example json body:

{
  "title": "My alert",
  "ruleId": 1,
  "ruleName": "Load peaking!",
  "ruleUrl": "http://url.to.grafana/db/dashboard/my_dashboard?panelId=2",
  "state": "alerting",
  "imageUrl": "http://s3.image.url",
  "message": "Load is peaking. Make sure the traffic is real and spin up more webfronts",
  "evalMatches": [
    {
      "metric": "requests",
      "tags": {},
      "value": 122
    }
  ]
}

state - The possible values for alert state are: ok, paused, alerting, pending, no_data.

DingDing/DingTalk

Instructions in Chinese.

In DingTalk PC Client:

Click “more” icon on upper right of the panel.
Click “Robot Manage” item in the pop menu, there will be a new panel call “Robot Manage”.
In the “Robot Manage” panel, select “customized: customized robot with Webhook”.
In the next new panel named “robot detail”, click “Add” button.
In “Add Robot” panel, input a nickname for the robot and select a “message group” which the robot will join in. click “next”.
There will be a Webhook URL in the panel, looks like this: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx. Copy this URL to the grafana Dingtalk setting page and then click “finish”.

Dingtalk supports the following “message type”: text, link and markdown. Only the link message type is supported.

Kafka

Notifications can be sent to a Kafka topic from Grafana using the Kafka REST Proxy. There are a couple of configuration options which need to be set up in Grafana UI under Kafka Settings:

Kafka REST Proxy endpoint.
Kafka Topic.

Once these two properties are set, you can send the alerts to Kafka for further processing or throttling.

Google Hangouts Chat

Notifications can be sent by setting up an incoming webhook in Google Hangouts chat. Configuring such a webhook is described here.

All supported notifiers

Name	Type	Supports images	Support alert rule tags
DingDing	`dingding`	yes, external only	no
Discord	`discord`	yes	no
Email	`email`	yes	no
Google Hangouts Chat	`googlechat`	yes, external only	no
Hipchat	`hipchat`	yes, external only	no
Kafka	`kafka`	yes, external only	no
Line	`line`	yes, external only	no
Microsoft Teams	`teams`	yes, external only	no
OpsGenie	`opsgenie`	yes, external only	no
Pagerduty	`pagerduty`	yes, external only	no
Prometheus Alertmanager	`prometheus-alertmanager`	yes, external only	yes
Pushover	`pushover`	yes	no
Sensu	`sensu`	yes, external only	no
Slack	`slack`	yes	no
Telegram	`telegram`	yes	no
Threema	`threema`	yes, external only	no
VictorOps	`victorops`	yes, external only	no
Webhook	`webhook`	yes, external only	yes

Enable images in notifications

Grafana can render the panel associated with the alert rule as a PNG image and include that in the notification. Read more about the requirements and how to configure image rendering here.

Most Notification Channels require that this image be publicly accessible (Slack and PagerDuty for example). In order to include images in alert notifications, Grafana can upload the image to an image store. It currently supports Amazon S3, Webdav, Google Cloud Storage and Azure Blob Storage. So to set that up you need to configure the external image uploader in your grafana-server ini config file.

Be aware that some notifiers requires public access to the image to be able to include it in the notification. So make sure to enable public access to the images. If you’re using local image uploader, your Grafana instance need to be accessible by the internet.

Notification services which need public image access are marked as ’external only'.

Use alert rule tags in notifications

Only available in Grafana v6.3+.

Grafana can include a list of tags (key/value) in the notification. It’s called alert rule tags to contrast with tags parsed from timeseries. It currently supports only the Prometheus Alertmanager notifier.

This is an optional feature. You can get notifications without using alert rule tags.

Configure the link back to Grafana from alert notifications

All alert notifications contain a link back to the triggered alert in the Grafana instance. This url is based on the domain setting in Grafana.