Alerting on Grafana Labs

Upgrade to Grafana Alerting

Sun, 12 Apr 2026 12:30:02 +0000

Upgrade to Grafana Alerting

Grafana Alerting is enabled by default for new installations or existing installations whether or not legacy alerting is configured.

Note: When upgrading, your dashboard alerts are migrated to a new format. This migration can be rolled back easily by opting out. If you have any questions regarding this migration, please contact us.

Existing installations that do not use legacy alerting will have Grafana Alerting enabled by default unless alerting is disabled in the configuration.

Likewise, existing installations that use legacy alerting will be automatically upgraded to Grafana Alerting unless you have opted out of Grafana Alerting before migration takes place. During the upgrade, legacy alerts are migrated to the new alerts type and no alerts or alerting data are lost.

Once the upgrade has taken place, you still have the option to roll back to legacy alerting. However, we do not recommend choosing this option. If you do choose to roll back, Grafana will restore your alerts to the alerts you had at the point in time when the upgrade took place. All new alerts and changes made exclusively in Grafana Alerting will be deleted.

Note: Cloud customers, who do not want to upgrade to Grafana Alerting, should contact customer support.

If you have opted out or rolled back, you can always choose to opt in to Grafana Alerting at a later point in time.

The following table provides details on the upgrade for Cloud, Enterprise, and OSS installations and the new Grafana Alerting UI.

Grafana instance upgraded to 9.0
Cloud	Existing Cloud installations with legacy dashboard alerting will have two alerting icons in the left navigation panel - the old alerting plugin icon and the new Grafana Alerting icon. During upgrade, existing alerts from the Cloud alerting plugin are migrated to Grafana Alerting. Once migration is complete, you can access and manage the older alerts from the new alerting Grafana Alerting icon in the navigation panel. The (older) Cloud alerting plugin is uninstalled from your cloud instance. Contact customer support if you do not wish to migrate to Grafana Alerting for your Cloud stack. If you choose to use legacy alerting, use the You will see the new Grafana Alerting icon as well as the old Cloud alerting plugin in the left navigation panel.
Enterprise	Existing Enterprise instances using legacy alerting will have both the old (marked as legacy) and the new alerting icons in the navigation panel. During upgrade, existing legacy alerts are migrated to Grafana Alerting. If you wish, you can opt-out of Grafana Alerting and roll back to legacy alerting. In that case, you can manage your legacy alerts from the alerting icon marked as legacy.
OSS	Existing OSS installations with legacy dashboard alerting will have two alerting icons in the left navigation panel - the old alerting icon (marked as legacy) and the new Grafana Alerting icon. During upgrade, existing legacy alerts are migrated to Grafana Alerting. If you wish, you can opt-out of Grafana Alerting and roll back to legacy alerting. In that case, you can manage your legacy alerts from the alerting icon marked as legacy.

Note: Legacy alerting will be deprecated in a future release (v10).

Alerting fundamentals

Sun, 12 Apr 2026 12:30:02 +0000

Alerting fundamentals

This section includes the following fundamental concepts of Grafana Alerting:

Create and manage rules

Sun, 12 Apr 2026 12:30:02 +0000

Create and manage Grafana Alerting rules

An alerting rule is a set of evaluation criteria that determines whether an alert will fire. The rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and optionally, the duration over which the condition is met.

While queries and expressions select the data set to evaluate, a condition sets the threshold that an alert must meet or exceed to create an alert. An interval specifies how frequently an alerting rule is evaluated. Duration, when configured, indicates how long a condition must be met. The rules can also define alerting behavior in the absence of data.

You can:

Contact points

Sun, 12 Apr 2026 12:30:02 +0000

Contact points

Use contact points to define how your contacts are notified when an alert fires. A contact point can have one or more contact point types, for example, email, slack, webhook, and so on. When an alert fires, a notification is sent to all contact point types listed for a contact point. Optionally, use message templates to customize notification messages for the contact point types.

You can configure Grafana managed contact points as well as contact points for an external Alertmanager data source. For more information, see Alertmanager.

Before you begin, see Grafana Alerting which explains the various components of Grafana Alerting. We also recommend that you familiarize yourself with some of the fundamental concepts of Grafana Alerting.

Notification policies

Sun, 12 Apr 2026 12:30:02 +0000

Notification policies

Notification policies determine how alerts are routed to contact points. Policies have a tree structure, where each policy can have one or more child policies. Each policy, except for the root policy, can also match specific alert labels. Each alert is evaluated by the root policy and subsequently by each child policy. If you enable the Continue matching subsequent sibling nodes option is enabled for a specific policy, then evaluation continues even after one or more matches. A parent policy’s configuration settings and contact point information govern the behavior of an alert that does not match any of the child policies. A root policy governs any alert that does not match a specific policy.

You can configure Grafana managed notification policies as well as notification policies for an external Alertmanager data source. For more information, see Alertmanager.

Grouping

Grouping is a new and key concept of Grafana Alerting that categorizes alert notifications of similar nature into a single funnel. This allows you to properly route alert notifications during larger outages when many parts of a system fail at once causing a high number of alerts to fire simultaneously.

For example, suppose you have 100 services connected to a database in different environments. These services are differentiated by the label env=environmentname. An alert rule is in place to monitor whether your services can reach the database named alertname=DatabaseUnreachable.

When a network partition occurs, half of your services can no longer reach the database. As a result, 50 different alerts (assuming half of your services) are fired. For this situation, you want to receive a single-page notification (as opposed to 50) with a list of the environments that are affected.

You can configure grouping to be group_by: [alertname] (take note that the env label is omitted). With this configuration in place, Grafana sends a single compact notification that has all the affected environments for this alert rule.

Note: Grafana also has a special label named ... that you can use to group all alerts by all labels (effectively disabling grouping), therefore each alert will go into its own group. It is different from the default of group_by: null where all alerts go into a single group.

Edit root notification policy

Note: Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organisations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 Alerts only if they have one organisation. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Notification policies.
From the Alertmanager dropdown, select an external Alertmanager. By default, the Grafana Alertmanager is selected.
In the Root policy section, click Edit (pen icon).
In Default contact point, update the contact point to whom notifications should be sent for rules when alert rules do not match any specific policy.
In Group by, choose labels to group alerts by. If multiple alerts are matched for this policy, then they are grouped by these labels. A notification is sent per group. If the field is empty (default), then all notifications are sent in a single group. Use a special label ... to group alerts by all labels (which effectively disables grouping).
In Timing options, select from the following options:
- Group wait Time to wait to buffer alerts of the same group before sending an initial notification. Default is 30 seconds.
- Group interval Minimum time interval between two notifications for a group. Default is 5 minutes.
- Repeat interval Minimum time interval for re-sending a notification if no new alerts were added to the group. Default is 4 hours.
Click Save to save your changes.

Add new specific policy

In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
Click Notification policies.
From the Alertmanager dropdown, select an Alertmanager. By default, the Grafana Alertmanager is selected.
To add a top level specific policy, go to the Specific routing section and click New specific policy.
In Matching labels section, add one or more rules for matching alert labels. For more information, see “Labels and label matchers”.
In Contact point, add the contact point to send notification to if alert matches only this specific policy and not any of the nested policies.
Optionally, enable Continue matching subsequent sibling nodes to continue matching nested policies even after the alert matched the parent policy. When this option is enabled, you can get more than one notification. Use it to send notification to a catch-all contact point as well as to one of more specific contact points handled by nested policies.
Optionally, enable Override grouping to specify the same grouping as the root policy. If this option is not enabled, the root policy grouping is used.
Optionally, enable Override general timings to override the timing options configured in the group notification policy.
Click Save policy to save your changes.

Add nested policy

Expand the specific policy you want to update.
Click Add nested policy, then add the details using information in Add new specific policy.
Click Save policy to save your changes.

Edit specific policy

In the Alerting page, click Notification policies to open the page listing existing policies.
Find the policy you want to edit, then click Edit (pen icon).
Make any changes using instructions in Add new specific policy.
Click Save policy.

Example

An example of an alert configuration.

Create a “default” contact point for slack notifications, and set it on root policy.
Edit the root policy grouping to group alerts by cluster, namespace and severity so that you get a notification per alert rule and specific kubernetes cluster and namespace.
Create specific route for alerts coming from the development cluster with an appropriate contact point.
Create a specific route for alerts with “critical” severity with a more invasive contact point type, like pager duty notification.
Create specific routes for particular teams that handle their own onduty rotations.

Alert groups

Sun, 12 Apr 2026 12:30:02 +0000

Alert groups

Alert groups show grouped alerts from an Alertmanager instance. By default, the alerts are grouped by the label keys for the root policy in notification policies. Grouping common alerts into a single alert group prevents duplicate alerts from being fired.

For more information, see:

Alerting high availability

Sun, 12 Apr 2026 12:30:02 +0000

About alerting high availability

The Grafana Alerting system has two main components: a Scheduler and an internal Alertmanager. The Scheduler evaluates your alert rules, while the internal Alertmanager manages routing and grouping.

When running Grafana Alerting in high availability, the operational mode of the scheduler remains unaffected, and each Grafana instance evaluates all alerts. The operational change happens in the Alertmanager when it deduplicates alert notifications across Grafana instances.

The coordination between Grafana instances happens via a Gossip protocol. Alerts are not gossiped between instances and each scheduler delivers the same volume of alerts to each Alertmanager.

The two types of messages gossiped between Grafana instances are:

Notification logs: Who (which instance) notified what (which alert).
Silences: If an alert should fire or not.

The notification logs and silences are persisted in the database periodically and during a graceful Grafana shut down.

For configuration instructions, refer to enable alerting high availability.

Silences

Sun, 12 Apr 2026 12:30:02 +0000

About alerting silences

Use silences to stop notifications from one or more alerting rules. Silences do not prevent alert rules from being evaluated. Nor do they not stop alerting instances from being shown in the user interface. Silences only stop notifications from getting created. A silence lasts for only a specified window of time.

You can configure Grafana managed silences as well as silences for an external Alertmanager data source. For more information, see Alertmanager.

Limitations

Sun, 12 Apr 2026 12:30:02 +0000

Limitations

Limited rule sources support

Grafana Alerting can retrieve alerting and recording rules stored in most available Prometheus, Loki, Mimir, and Alertmanager compatible data sources.

It does not support reading or writing alerting rules from any other data sources but the ones previously mentioned at this time.

Prometheus version support

We support the latest two minor versions of both Prometheus and Alertmanager. We cannot guarantee that older versions will work.

As an example, if the current Prometheus version is 2.31.1, we support >= 2.29.0.

Grafana is not an alert receiver

Grafana is not an alert receiver; is it an alert generator. This means that Grafana cannot receive alerts from anything other than its internal alert generator.

Receiving alerts from Prometheus (or anything else) is not supported at the time.

For more information, refer to this GitHub discussion.

Performance considerations

Sun, 12 Apr 2026 12:30:02 +0000

Alerting performance considerations

Grafana Alerting supports multi-dimensional alerting, where one alert rule can generate many alerts. For example, you can configure an alert rule to fire an alert every time the CPU of individual VMs max out. This topic discusses performance considerations resulting from multi-dimensional alerting.

Evaluating alerting rules consumes RAM and CPU to compute the output of an alerting query, and network resources to send alert notifications and write the results to the Grafana SQL database. The configuration of individual alert rules affects the resource consumption and, therefore, the maximum number of rules a given configuration can support.

The following section provides a list of alerting performance considerations.

Frequency of rule evaluation consideration. The “Evaluate Every” property of an alert rule controls the frequency of rule evaluation. We recommend using the lowest acceptable evaluation frequency to support more concurrent rules.
Cardinality of the rule’s result set. For example, suppose you are monitoring API response errors for every API path, on every VM in your fleet. This set has a cardinality of n number of paths multiplied by v number of VMs. You can reduce the cardinality of a result set - perhaps by monitoring errors-per-VM instead of for each path per VM.
Complexity of the alerting query consideration. Queries that data sources can process and respond to quickly consume fewer resources. Although this consideration is less important than the other considerations listed above, if you have reduced those as much as possible, looking at individual query performance could make a difference.

Each evaluation of an alert rule generates a set of alert instances; one for each member of the result set. The state of all the instances is written to the alert_instance table in Grafana’s SQL database.

Grafana Alerting exposes a metric, grafana_alerting_rule_evaluations_total that counts the number of alert rule evaluations. To get a feel for the influence of rule evaluations on your Grafana instance, you can observe the rate of evaluations and compare it with resource consumption. In a Prometheus-compatible database, you can use the query rate(grafana_alerting_rule_evaluations_total[5m]) to compute the rate over 5 minute windows of time. It’s important to remember that this isn’t the full picture of rule evaluation. For example, the load will be unevenly distributed if you have some rules that evaluate every 10 seconds, and others every 30 minutes.

These factors all affect the load on the Grafana instance, but you should also be aware of the performance impact that evaluating these rules has on your data sources. Alerting queries are often the vast majority of queries handled by monitoring databases, so the same load factors that affect the Grafana instance affect them as well.

Images in notifications

Sun, 12 Apr 2026 12:30:02 +0000

Images in notifications

Images in notifications helps recipients of alert notifications better understand why an alert has fired or resolved by including an image of the panel associated with the Grafana managed alert rule.

Note: Images in notifications are not available for Grafana Mimir and Loki managed alert rules, or when Grafana is set up to send alert notifications to an external Alertmanager.

If Grafana is set up to send images in notifications, it takes a screenshot of the panel for the Grafana managed alert rule when either of the following happen:

The alert rule transitions from pending to firing
The alert rule transitions from firing to OK

Grafana does not support images for alert rules that are not associated with a panel. An alert rule is associated with a panel when it has both Dashboard UID and Panel ID annotations.

Images are stored in the data path and so Grafana must have write-access to this path. If Grafana cannot write to this path then screenshots cannot be saved to disk and an error will be logged for each failed screenshot attempt. In addition to storing images on disk, Grafana can also store the image in an external image store such as Amazon S3, Azure Blob Storage, Google Cloud Storage and even Grafana where screenshots are stored in public/img/attachments. Screenshots older than temp_data_lifetime are deleted from disk but not the external image store. If Grafana is the external image store then screenshots are deleted from data but not from public/img/attachments.

Note: It is recommended that you use an external image store, as not all contact points support uploading images from disk. It is also possible that the image on disk is deleted before an alert notification is sent if temp_data_lifetime is less than the group_wait and group_interval options used in Alertmanager.

Requirements

To use images in notifications, Grafana must be set up to use image rendering. It is also recommended that Grafana is set up to upload images to an external image store such as Amazon S3, Azure Blob Storage, Google Cloud Storage or even Grafana.

Configuration

If Grafana has been set up to use image rendering images in notifications can be turned on via the capture option in [unified_alerting.screenshots]:

# Enable screenshots in notifications. This option requires the Grafana Image Renderer plugin.
# For more information on configuration options, refer to [rendering].
capture = true

It is recommended that max_concurrent_screenshots is set to a value that is less than or equal to concurrent_render_request_limit. The default value for both max_concurrent_screenshots and concurrent_render_request_limit is 5:

# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5

If Grafana has been set up to use an external image store, upload_external_image_storage should be set to true:

# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false

Restart Grafana for the changes to take affect.

Supported notifiers

Images in notifications are supported in the following notifiers and additional support will be added in the future:

Name	Upload images from disk	Include images from URL
DingDing	No	No
Discord	Yes	Yes
Email	Yes	Yes
Google Hangouts Chat	No	Yes
Kafka	No	No
Line	No	No
Microsoft Teams	No	Yes
Opsgenie	No	Yes
Pagerduty	No	Yes
Prometheus Alertmanager	No	No
Pushover	Yes	No
Sensu Go	No	No
Slack	No	Yes
Telegram	No	No
Threema	No	No
VictorOps	No	No
Webhook	No	Yes

Include images from URL refers to using the external image store.

Metrics

Grafana provides the following metrics to observe the performance and failure rate of images in notifications. For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter grafana_screenshot_failures_total is updated.

grafana_screenshot_cache_hits_total
grafana_screenshot_cache_misses_total
grafana_screenshot_duration_seconds
grafana_screenshot_failures_total
grafana_screenshot_successes_total
grafana_screenshot_upload_failures_total
grafana_screenshot_upload_successes_total

Limitations

Images in notifications are not available for Grafana Mimir and Loki managed alert rules, or when Grafana is set up to send alert notifications to an external Alertmanager.
When alerts generated by different alert rules are sent in a single notification, there may be screenshots for each alert rule. This happens if an alert group contains multiple alerting rules. The order the images are attached is random. If you need to guarantee the ordering of images, make sure that your alert groups contain a single alerting rule.
Some contact points only handle a single image. In this case, the first image associated with an alert will be attached. Because the ordering is random, this may not always be an image for the same alert rule. If you need to guarantee you receive a screenshot for a particular rule, make sure that your alert groups contain a single alerting rule.