This is documentation for the next version of Grafana documentation. For the latest stable release, go to the latest version.
Grafana documentation
Alerting
Configure alert rules
Configure Grafana-managed alert rulesConfigure Grafana-managed alert rules
Grafana-managed alert rules are the default way to create alert rules in Grafana.
Grafana-managed rules inherit their model from Prometheus Alerting and extend it with greater flexibility—such as multi-data source queries, expression-based transformations, advanced alert conditions, images in notifications, custom states, and more.
To create or edit Grafana-managed alert rules, follow the instructions below.
Tip
For quick-start tutorials on key alerting features, see Getting started with Grafana Alerting tutorials.
Before you begin
Before you create Grafana-managed alert rules, review the following requirements and options.
Supported data sources
Grafana-managed alert rules can query backend data sources when the data source’s plugin.json file sets {"backend": true, "alerting": true}.
Before you create an alert rule, verify that the data sources you plan to query are compatible and properly configured.
You can find the public data sources that support alert rules in the Grafana Plugins directory.
Alert rule limits in Grafana Cloud
In Grafana Cloud, the number of Grafana-managed alert rules you can create depends on your Grafana Cloud plan.
- Free Forever plan: You can create up to 100 free alert rules, with each alert rule having a maximum of 1000 alert instances.
- All paid plans: They have a soft limit of 2000 alert rules and support unlimited alert instances. To increase the limit, open a support ticket from the Cloud portal.
Permissions
Only users with Edit permissions for the folder storing the rules can edit or delete Grafana-managed alert rules. Only admins can restore deleted Grafana-managed alert rules.
Provisioning
Note that if you delete an alert resource created in the UI, you can no longer retrieve it.
To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.
Default versus Advanced options
Grafana Alerting provides multiple configuration options to support different alerting needs.
To simplify alert rule creation, some settings display a Default and an Advanced configuration view:
Use Default to create basic query conditions and route alerts directly to a contact point. Use Advanced for more complex query definitions, or to route alerts using notification policies.
You can switch between options at any time. After you create an alert rule, Grafana remembers your last selection and uses it as the default for the next rule.
If you prefer to always use advanced configuration, you can disable the Default modes by disabling the alertingQueryAndExpressionsStepMode and alertingNotificationsStepMode feature toggles.
Set alert rule name
Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.
This name is displayed in the alert rule list. It is also the
alertnamelabel for every alert instance that is created from this rule.
Define query and condition
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
The Default option allows to configure one query and one alert condition. The Advanced option allows multiple queries and expressions for more complex rule definitions.
Select a data source.
Add a query.
Add an alert condition.
The When input includes the reducer function and the last input is the threshold.
Click Preview to verify.
Select a data source.
From the Options drop-down menu, specify a time range.
Note that Grafana Alerting only supports fixed relative time ranges, for example,
now-24hr: now. It doesn’t support absolute time ranges:2021-12-02 00:00:00 to 2021-12-05 23:59:592or semi-relative time ranges:now/d to: now.Add a query.
To add multiple queries, click Add query.
All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click Switch to data source-managed alert rule.
Add one or more expressions.
a. For each expression, select either Classic condition to create a single alert rule, or choose from the Math, Reduce, and Resample options to generate separate alert for each series.
When using Prometheus, you can use an instant vector and built-in functions, so you don’t need to add additional expressions.
b. Click Preview to verify that the expression is successful.
To add a recovery threshold, enable the Custom recovery threshold toggle and enter a value that defines when the alert should recover—transition to
Normalstate from theAlertingorPendingstate.You can only add one recovery threshold, and it must be part of the alert condition.
Click Set as alert condition on the query or expression you want to set as your alert condition.
You can switch between Default and Advanced options at any time.
Switching from Advanced to Default may result in queries and expressions that can’t be converted. In this case, a warning message asks if you want to continue to reset to default settings.
If the Default versus Advanced options feature is not enabled in your Grafana instance, follow the Advanced options instructions.
Set folder and labels
Organize your alert rule with a folder and set of labels.
In the Labels section, you can optionally choose whether to add labels to organize your alert rules and their notifications. For more details, refer to alert rule labels.
Select a folder or click + New folder.
Add labels, if required.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Configure alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated concurrently over the same time interval.
Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
After a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Optionally, set the Keep firing for period.
You can set the minimum amount of time that an alert remains firing after the breached threshold expression no longer returns any results. This sets an alert to a “Recovering” state for the duration of time set here. The Recovering state can be used to reduce noise from flapping alerts. Select “none” stop an alert from firing immediately after the breach threshold is cleared.
Turn on pause alert notifications, if required.
You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and doesn’t create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.
In Configure no data and error handling, you can define the alerting behavior and alerting state for two scenarios:
- When the evaluation returns No data or all values are null.
- When the evaluation returns Error or timeout.
Configure no data and error handling
For more details, refer to alert instance states and modify the no data or error state.
In Configure no data and error handling, you can also configure Missing series evaluations to resolve: how many consecutive evaluation intervals must pass without data before an alert instance is considered stale.
Configure notifications
Configure who receives notifications when the alert rule fires.
The Default option allows to select a contact point to handle notifications for this alert rule. The Advanced option routes notifications through notification policies.
Select an existing contact point.
All notifications for this alert rule are sent to this contact point automatically and notification policies aren’t used.
Optionally, configure additional notification settings like grouping or timing options.
With this option, all notifications for this alert rule are managed by the notification policy tree, which routes alerts based on their labels.
You can preview which notification policy would handle notifications from this alert rule.
Click Preview routing after configuring the previous alert rule settings.
Based on the alert labels and query labels, Grafana displays the notification policies that would handle the notifications.
Click View route to view policy and routing details.
You can switch between Default and Advanced options at any time.
If the Default versus Advanced options feature is not enabled in your Grafana instance, follow the Advanced options instructions.
Configure notification message
Use annotations to add information to alert messages that can help respond to the alert.
Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.
Grafana provides several optional annotations.
Optional: Add a summary.
Short summary of what happened and why.
Optional: Add a description.
Description of what the alert rule does.
Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
Optional: Add a custom annotation.
Add any additional information that could help address the alert.
Optional: Link dashboard and panel.
Link the alert rule to a panel to facilitate alert investigation.
Click Save rule.
