Configure Grafana-managed alert rules
Grafana-managed rules are the most flexible alert rule type. They allow you to create alerts that can act on data from any of our supported data sources. In addition to supporting multiple data sources, you can also add expressions to transform your data and set alert conditions. Using images in alert notifications is also supported. This is the only type of rule that allows alerting from multiple data sources in a single rule definition.
Multiple alert instances can be created as a result of one alert rule (also known as a multi-dimensional alerting).
Note
For Grafana Cloud Free Forever, you can create up to 100 free Grafana-managed alert rules with each alert rule having a maximum of 1000 alert instances.
Grafana-managed alert rules can only be edited or deleted by users with Edit permissions for the folder storing the rules.
If you delete an alert resource created in the UI, you can no longer retrieve it. To make a backup of your configuration and to be able to restore deleted alerting resources, create your alerting resources using file provisioning, Terraform, or the Alerting API.
Before you begin
You can use default or advanced options for Grafana-managed alert rule creation. The default options streamline rule creation with a cleaner header and a single query and condition. For more complex rules, use advanced options to add multiple queries and expressions.
Default and advanced options are enabled by default for Grafana Cloud users and this feature is being rolled out progressively.
For OSS users,enable the alertingQueryAndExpressionsStepMode
feature toggle.
Note
Once you have created an alert rule using one of the options, the system defaults to this option for the next alert rule you create.
You can toggle between the two options. However, if you want to switch from advanced options to the default, it may be that your query and expressions cannot be converted. In this case, a warning message checks whether you want to continue to reset to default settings.
Steps
To create a Grafana-managed alert rule, use the in-product alert creation flow and follow these steps.
To get started quickly, refer to our tutorial on getting started with Grafana alerting.
Set alert rule name
Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.
This name is displayed in the alert rule list. It is also the
alertname
label for every alert instance that is created from this rule.
Define query and condition
Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.
Add a query.
Add an alert condition.
The When input includes the reducer function and the last input is the threshold.
Click Preview to verify.
- Select a data source.
- From the Options dropdown, specify a time range.
Note
Grafana Alerting only supports fixed relative time ranges, for example,
now-24hr: now
.It does not support absolute time ranges:
2021-12-02 00:00:00 to 2021-12-05 23:59:592
or semi-relative time ranges:now/d to: now
.
Add a query.
To add multiple queries, click Add query.
All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click Switch to data source-managed alert rule.
Add one or more expressions.
a. For each expression, select either Classic condition to create a single alert rule, or choose from the Math, Reduce, and Resample options to generate separate alert for each series.
Note
When using Prometheus, you can use an instant vector and built-in functions, so you don’t need to add additional expressions.b. Click Preview to verify that the expression is successful.
To add a recovery threshold, turn the Custom recovery threshold toggle on and fill in a value for when your alert rule should stop firing.
You can only add one recovery threshold in a query and it must be the alert condition.
Click Set as alert condition on the query or expression you want to set as your alert condition.
Set alert evaluation behavior
Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.
To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.
Select a folder or click + New folder.
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated concurrently over the same time interval.
Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.
Turn on pause alert notifications, if required.
Note
You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and doesn’t create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.In Configure no data and error handling, configure alerting behavior in the absence of data.
Use the guidelines in No data and error handling.
Configure labels and notifications
In the Labels section, you can optionally choose whether to add labels to organize your alert rules, make searching easier, as well as set which notification policy should handle your firing alert instance.
In the Configure notifications section, you can choose to select a contact point directly from the alert rule form or choose to use notification policy routing as well as set up mute timings and groupings.
Complete the following steps to set up labels and notifications.
Add labels, if required.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.
Configure who receives a notification when an alert rule fires by either choosing Select contact point or Use notification policy.
Select contact point
Choose this option to select an existing contact point.
All notifications for this alert rule are sent to this contact point automatically and notification policies are not used.
You can also optionally select a mute timing as well as groupings and timings to define when not to send notifications.
Note
An auto-generated notification policy is generated. Only admins can view these auto-generated policies from the Notification policies list view. Any changes have to be made in the alert rules form.
Use notification policy
Choose this option to use the notification policy tree to direct your notifications.
Note
All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.Preview your alert instance routing set up.
Based on the labels added, alert instances are routed to the following notification policies displayed.
Expand each notification policy below to view more details.
Click See details to view alert routing details and an email preview.
Add annotations
Add annotations. to provide more context on the alert in your alert notification message.
Annotations add metadata to provide more information on the alert in your alert notification message. For example, add a Summary annotation to tell you which value caused the alert to fire or which server it happened on.
Optional: Add a summary.
Short summary of what happened and why.
Optional: Add a description.
Description of what the alert rule does.
Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
Optional: Add a custom annotation
Optional: Add a dashboard and panel link.
Links alert rules to panels in a dashboard.
Note
At the moment, alert rules are only supported in time series and alert list visualizations.Click Save rule.
Configure no data and error handling
In Configure no data and error handling, you can define the alerting behavior when the evaluation returns no data or an error.
For details about alert states, refer to lifecycle of alert instances.
You can configure the alert instance state when its evaluation returns no data:
No Data configuration | Description |
---|---|
No Data | The default option. Sets alert instance state to No data .The alert rule also creates a new alert instance DatasourceNoData with the name and UID of the alert rule, and UID of the datasource that returned no data as labels. |
Alerting | Sets the alert instance state to Pending and then transitions to Alerting once the pending period ends. If you sent the pending period to 0, the alert instance state is immediately set to Alerting . |
Normal | Sets alert instance state to Normal . |
Keep Last State | Maintains the alert instance in its last state. Useful for mitigating temporary issues, refer to Keep last state. |
You can also configure the alert instance state when its evaluation returns an error or timeout.
Error configuration | Description |
---|---|
Error | The default option. Sets alert instance state to Error .The alert rule also creates a new alert instance DatasourceError with the name and UID of the alert rule, and UID of the datasource that returned no data as labels. |
Alerting | Sets alert instance state to Alerting . It transitions from Pending to Alerting after the pending period has finished. |
Normal | Sets alert instance state to Normal . |
Keep Last State | Maintains the alert instance in its last state. Useful for mitigating temporary issues, refer to Keep last state. |
When you configure the No data or Error behavior to Alerting
or Normal
, Grafana will attempt to keep a stable set of fields under notification Values
. If your query returns no data or an error, Grafana re-uses the latest known set of fields in Values
, but will use -1
in place of the measured value.