Configure Alerting on Grafana Labs

Configure Grafana-managed alert rules

Mon, 16 Mar 2026 04:58:00 +0000

Configure Grafana-managed alert rules

Grafana-managed rules are the most flexible alert rule type. They allow you to create alerts that can act on data from any of our supported data sources. In addition to supporting multiple data sources, you can also add expressions to transform your data and set alert conditions. Using images in alert notifications is also supported. This is the only type of rule that allows alerting from multiple data sources in a single rule definition.

Multiple alert instances can be created as a result of one alert rule (also known as a multi-dimensional alerting).

Note
For Grafana Cloud, there are limits on how many Grafana-managed alert rules you can create. These are as follows:

Free: 100 alert rules

Paid: 2000 alert rules

Grafana managed alert rules can only be edited or deleted by users with Edit permissions for the folder storing the rules.

If you delete an alerting resource created in the UI, you can no longer retrieve it. To make a backup of your configuration and to be able to restore deleted alerting resources, create your alerting resources using file provisioning, Terraform, or the Alerting API.

Watch this video to learn more about creating alert rules:

{ responded = true; response && response.status === 200 ? vimeo_is_up = true : vimeo_is_up = false; }) .catch(error => { responded = true; })" >

In the following sections, we’ll guide you through the process of creating your Grafana-managed alert rules.

To create a Grafana-managed alert rule, use the in-product alert creation flow and follow these steps to help you.

Set alert rule name

Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.

This name is displayed in the alert rule list. It is also the alertname label for every alert instance that is created from this rule.

Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

Select a data source.
From the Options dropdown, specify a time range.

Note:

Grafana Alerting only supports fixed relative time ranges, for example, now-24hr: now.

It does not support absolute time ranges: 2021-12-02 00:00:00 to 2021-12-05 23:59:592 or semi-relative time ranges: now/d to: now.
Add a query.

To add multiple queries, click Add query.

All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click Switch to data source-managed alert rule.
Add one or more expressions.

a. For each expression, select either Classic condition to create a single alert rule, or choose from the Math, Reduce, and Resample options to generate separate alert for each series.

Note
When using Prometheus, you can use an instant vector and built-in functions, so you don’t need to add additional expressions.

b. Click Preview to verify that the expression is successful.

Note
The recovery threshold feature is currently only available in OSS.

To add a recovery threshold, turn the Custom recovery threshold toggle on and fill in a value for when your alert rule should stop firing.

You can only add one recovery threshold in a query and it must be the alert condition.
Click Set as alert condition on the query or expression you want to set as your alert condition.

Set alert evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.

Select a folder or click + New folder.
Select an evaluation group or click + New evaluation group.

If you are creating a new evaluation group, specify the interval for the group.

All rules within the same group are evaluated concurrently over the same time interval.
Enter a pending period.

The pending period is the period in which an alert rule can be in breach of the condition until it fires.

Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Turn on pause alert notifications, if required.

Note
You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and doesn’t create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.
In Configure no data and error handling, configure alerting behavior in the absence of data.

Use the guidelines in No data and error handling.

Add annotations

Add annotations. to provide more context on the alert in your alert notifications.

Annotations add metadata to provide more information on the alert in your alert notifications. For example, add a Summary annotation to tell you which value caused the alert to fire or which server it happened on.

[Optional] Add a summary.

Short summary of what happened and why.
[Optional] Add a description.

Description of what the alert rule does.
[Optional] Add a Runbook URL.

Webpage where you keep your runbook for the alert
[Optional] Add a custom annotation
[Optional] Add a dashboard and panel link.

Links alerts to panels in a dashboard.

Configure notifications

Add labels to your alert rules to set which notification policy should handle your firing alert instances.

All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.

Add labels if you want to change the way your notifications are routed.

Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Preview your alert instance routing set up.

Based on the labels added, alert instances are routed to the following notification policies displayed.

Expand each notification policy below to view more details.
Click See details to view alert routing details and an email preview.
Click Save rule.

Single and multi-dimensional rule

For Grafana managed alerts, you can create a rule with a classic condition or you can create a multi-dimensional rule.

Rule with classic condition

Use the classic condition expression to create a rule that triggers a single alert when its condition is met. For a query that returns multiple series, Grafana does not track the alert state of each series. As a result, Grafana sends only a single alert even when alert conditions are met for multiple series.

For more information, see expressions documentation.

Multi-dimensional rule

To generate a separate alert for each series, create a multi-dimensional rule. Use Math, Reduce, or Resample expressions to create a multi-dimensional rule. For example:

Add a Reduce expression for each query to aggregate values in the selected time range into a single value. (Not needed for rules using numeric data.
Add a Math expression with the condition for the rule. Not needed in case a query or a reduce expression already returns 0 if rule should not fire, or a positive number if it should fire. Some examples: $B > 70 if it should fire in case value of B query/expression is more than 70. $B < $C * 100 in case it should fire if value of B is less than value of C multiplied by 100. If queries being compared have multiple series in their results, series from different queries are matched if they have the same labels or one is a subset of the other.

Note: Grafana does not support alert queries with template variables. More information is available at https://community.grafana.com/t/template-variables-are-not-supported-in-alert-queries-while-setting-up-alert/2514.

Configure no data and error handling

Configure alerting behavior when your alert rule evaluation returns no data or an error.

Note: Alert rules that are configured to fire when an evaluation returns no data or error only fire when the entire duration of the evaluation period has finished. This means that rather than immediately firing when the alert rule condition is breached, the alert rule waits until the time set as the For field has finished and then fires, reducing alert noise and allowing for temporary data availability issues.

If your alert rule evaluation returns no data, you can set the state on your alert rule to appear as follows:

No Data	Description
No Data	Creates a new alert `DatasourceNoData` with the name and UID of the alert rule, and UID of the datasource that returned no data as labels.
Alerting	Sets alert rule state to `Alerting`. The alert rule waits until the time set in the For field has finished before firing.
Ok	Sets alert rule state to `Normal`.

If your evaluation returns an error, you can set the state on your alert rule to appear as follows:

Error	Description
Error	Creates an alert instance `DatasourceError` with the name and UID of the alert rule, and UID of the datasource that returned no data as labels.
Alerting	Sets alert rule state to `Alerting`. The alert rule waits until the time set in the For field has finished before firing.
Ok	Sets alert rule state to `Normal`.

Resolve stale alert instances

An alert instance is considered stale if its dimension or series has disappeared from the query results entirely for two evaluation intervals.

Stale alert instances that are in the Alerting/NoData/Error states are automatically marked as Resolved and the grafana_state_reason annotation is added to the alert instance with the reason MissingSeries.

Create alerts from panels

Create alerts from any panel type. This means you can reuse the queries in the panel and create alerts based on them.

Navigate to a dashboard in the Dashboards section.
In the top right corner of the panel, click on the three dots (ellipses).
From the dropdown menu, select More… and then choose New alert rule.

This will open the alert rule form, allowing you to configure and create your alert based on the current panel’s query.

Configure data source-managed alert rules

Mon, 16 Mar 2026 04:58:00 +0000

Configure data source-managed alert rules

Create alert rules for an external Grafana Mimir or Loki instance that has ruler API enabled; these are called data source-managed alert rules.

Note:

Alert rules for an external Grafana Mimir or Loki instance can be edited or deleted by users with Editor or Admin roles.

Before you begin

Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alert rules.
For Grafana Mimir and Loki data sources, enable the Ruler API by configuring their respective services.
- Loki - The local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
- Grafana Mimir - use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.

Watch this video to learn more about how to create a Mimir managed alert rule:

{ responded = true; response && response.status === 200 ? vimeo_is_up = true : vimeo_is_up = false; }) .catch(error => { responded = true; })" >

Note
If you do not want to manage alert rules for a particular Loki or Prometheus data source, go to its settings and clear the Manage alerts via Alerting UI checkbox.

In the following sections, we’ll guide you through the process of creating your data source-managed alert rules.

To create a data source-managed alert rule, use the in-product alert creation flow and follow these steps to help you.

Set alert rule name

Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.

This name is displayed in the alert rule list. It is also the alertname label for every alert instance that is created from this rule.

Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

Note:

All alert rules are managed by Grafana by default. To switch to a data source-managed alert rule, click Switch to data source-managed alert rule.

Select a data source from the drop-down list.

You can also click Open advanced data source picker to see more options, including adding a data source (Admins only).
Enter a PromQL or LogQL query.
Click Preview alerts.

Set alert evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

Select a namespace or click + New namespace.
Select an evaluation group or click + New evaluation group.

If you are creating a new evaluation group, specify the interval for the group.

All rules within the same group are evaluated sequentially over the same time interval.
Enter a pending period.

The pending period is the period in which an alert rule can be in breach of the condition until it fires.

Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.

Add annotations

Add annotations. to provide more context on the alert in your alert notifications.

[Optional] Add a summary.

Short summary of what happened and why.
[Optional] Add a description.

Description of what the alert rule does.
[Optional] Add a Runbook URL.

Webpage where you keep your runbook for the alert
[Optional] Add a custom annotation
[Optional] Add a dashboard and panel link.

Links alerts to panels in a dashboard.

Configure notifications

Add labels to your alert rules to set which notification policy should handle your firing alert instances.

Add labels if you want to change the way your notifications are routed.

Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Click Save rule.

Configure recording rules

Mon, 16 Mar 2026 04:58:00 +0000

Configure recording rules

You can create and manage recording rules for an external Grafana Mimir or Loki instance. Recording rules calculate frequently needed expressions or computationally expensive expressions in advance and save the result as a new set of time series. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.

Note:

Recording rules are run as instant rules, which means that they run every 10s. To overwrite this configuration, update the min_interval in your custom configuration file.

min_interval sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules will be adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.

This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.

Before you begin

Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.
For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.
- Loki - The local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
- Grafana Mimir - use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.

Create recording rules

To create recording rules, follow these steps.

Click Alerts & IRM -> Alerting -> Alert rules.
Click the More dropdown and then New recording rule.
Set rule name.

The recording rule name must be a Prometheus metric name and contain no whitespace.
Define query.
- Select your Loki or Prometheus data source.
- Enter a query.
Add namespace and group.
- From the Namespace dropdown, select an existing rule namespace or add a new one. Namespaces can contain one or more rule groups and only have an organizational purpose.
- From the Group dropdown, select an existing group within the selected namespace or add a new one. Newly created rules are appended to the end of the group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.
Add labels.
- Add custom labels selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value .
Click Save rule to save the rule or Save rule and exit to save the rule and go back to the Alerting page.

Configure contact points

Mon, 16 Mar 2026 04:58:00 +0000

Configure contact points

Use contact points to define how your contacts are notified when an alert rule fires. You can add, edit, delete, and test a contact point.

Add a contact point

Complete the following steps to add a contact point.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Contact points.
From the Choose Alertmanager dropdown, select an Alertmanager. By default, Grafana Alertmanager is selected.
On the Contact Points tab, click + Add contact point.
Enter a descriptive name for the contact point.
From Integration, select a type and fill out mandatory fields. For example, if you choose email, enter the email addresses. Or if you choose Slack, enter the Slack channel(s) and users who should be contacted.
Some contact point integrations, like email or webhook, have optional settings. In Optional settings, specify additional settings for the selected contact point integration.
In Notification settings, optionally select Disable resolved message if you do not want to be notified when an alert resolves.
To add another contact point integration, click Add contact point integration and repeat steps 6 through 8.
Save your changes.

Edit a contact point

Complete the following steps to edit a contact point.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Contact points to view a list of existing contact points.
On the Contact Points tab, find the contact point you want to edit, and then click Edit.
Update the contact point and save your changes.

Delete a contact point

Complete the following steps to delete a contact point.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Contact points to view a list of existing contact points.
On the Contact Points tab, find the contact point you want to delete, and then click More -> Delete.
In the confirmation dialog, click Yes, delete.

Note
You cannot delete contact points that are in use by a notification policy. Either delete the notification policy or update it to use another contact point.

Test a contact point

Complete the following steps to test a contact point.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Contact points to view a list of existing contact points.
On the Contact Points tab, find the contact point you want to test, then click Edit. You can also create a new contact point if needed.
Click Test to open the contact point testing modal.
Choose whether to send a predefined test notification or choose custom to add your own custom annotations and labels to include in the notification.
Click Send test notification to fire the alert.

Configure notification policies

Mon, 16 Mar 2026 04:58:00 +0000

Configure notification policies

Notification policies determine how alerts are routed to contact points.

Policies have a tree structure, where each policy can have one or more nested policies. Each policy, except for the default policy, can also match specific alert labels.

Each alert is evaluated by the default policy and subsequently by each nested policy.

If the Continue matching subsequent sibling nodes option is enabled for a nested policy, then evaluation continues even after one or more matches. A parent policy’s configuration settings and contact point information govern the behavior of an alert that does not match any of the nested policies. A default policy governs any alert that does not match a nested policy.

You can configure Grafana-managed notification policies as well as notification policies for an external Alertmanager data source.

For more information on notification policies, see fundamentals of Notification Policies.

Edit default notification policy

Note
Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organizations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 Alerts only if they have one organization. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Notification policies.
From the Choose Alertmanager dropdown, select an external Alertmanager. By default, the Grafana Alertmanager is selected.
In the Default policy section, click … -> Edit.
In Default contact point, update the contact point to whom notifications should be sent for rules when alert rules do not match any specific policy.
In Group by, choose labels to group alerts by. If multiple alerts are matched for this policy, then they are grouped by these labels. A notification is sent per group. If the field is empty (default), then all notifications are sent in a single group. Use a special label ... to group alerts by all labels (which effectively disables grouping).
In Timing options, select from the following options:
- Group wait Time to wait to buffer alerts of the same group before sending an initial notification. Default is 30 seconds.
- Group interval Minimum time interval between two notifications for a group. Default is 5 minutes.
- Repeat interval Minimum time interval for re-sending a notification if no new alerts were added to the group. Default is 4 hours.
Click Save to save your changes.

Add new nested policy

To create a new notification policy, you need to follow its tree structure. New policies created on the trunk of the tree (default policy), are the tree branches. And, subsequently, each branch can bear their own child policies. This is why you will always be adding a new nested policy under either the default policy, or under a already nested policy.

In the left-side menu, click Alerts & IRM and then Alerting.
Click Notification policies.
From the Choose Alertmanager dropdown, select an Alertmanager. By default, the Grafana Alertmanager is selected.
To add a top level specific policy, go to the Specific routing section (either to the default policy, or to another existing policy in which you would like to add a new nested policy) and click +New nested policy.
In the Matching labels section, add one or more rules for matching alert labels.
In the Contact point dropdown, select the contact point to send notification to if alert matches only this specific policy and not any of the nested policies.
Optionally, enable Continue matching subsequent sibling nodes to continue matching sibling policies even after the alert matched the current policy. When this option is enabled, you can get more than one notification for one alert.
Optionally, enable Override grouping to specify the same grouping as the default policy. If this option is not enabled, the default policy grouping is used.
Optionally, enable Override general timings to override the timing options configured in the group notification policy.
Click Save policy to save your changes.

Add nested policy

In the left-side menu, click Alerts & IRM and then Alerting.
Click Notification policies.
Expand the specific policy you want to update.
Click + Add nested policy, then add the details using information in Add new specific policy.
Click Save policy to save your changes.

Edit notification policies

In the left-side menu, click Alerts & IRM, and then Alerting.
Click Notification policies.
Find the policy you want to edit, then click … -> Edit.
Make any changes using instructions in Add new specific policy.
Save your changes.

Searching for policies

Grafana allows you to search within the tree of policies by the following:

Label matchers
Contact Points

To search by contact point simply select a contact point from the Search by contact point dropdown. The policies that use that contact point will be highlighted in the user interface.

To search by label matchers simply enter a valid matcher in the Search by matchers input field. Multiple matchers can be combined with a comma (,).

An example of a valid matchers search input is:

severity=high, region=~EMEA|NASA

All matched policies will be exact matches, we currently do not support regex-style or partial matching.

Example

An example of an alert configuration.

Create a “default” contact point for slack notifications, and set it on the default policy.
Edit the default policy grouping to group alerts by cluster, namespace and severity so that you get a notification per alert rule and specific kubernetes cluster and namespace.
Create specific route for alerts coming from the development cluster with an appropriate contact point.
Create a specific route for alerts with “critical” severity with a more invasive contact point integration, like pager duty notification.
Create specific routes for particular teams that handle their own on-call rotations.