Documentation Grafana Cloud Alerts and IRM Alerting Configure alert rules Configure Grafana-managed alert rules

Configure Grafana-managed alert rules

Grafana-managed rules can query data from multiple data sources in a single alert rule. They are the most flexible alert rule type. You can also add expressions to transform your data, set alert conditions, and images in alert notifications.

To create or edit Grafana-managed alert rules, follow the instructions below. For a practical example, check out our tutorial on getting started with Grafana alerting.

Before you begin

Verify that the data sources you plan to query in the alert rule are compatible with Grafana-managed alert rules and are properly configured.

Permissions

Only users with Edit permissions for the folder storing the rules can edit or delete Grafana-managed alert rules.

Provisioning

Note that if you delete an alert resource created in the UI, you can no longer retrieve it.

To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.

Default vs Advanced options

You can use default or advanced options for Grafana-managed alert rule creation. The default options streamline rule creation with a cleaner header and a single query and condition. For more complex rules, use advanced options to add multiple queries and expressions.

You can toggle between the two options. Once you have created an alert rule, the system defaults to your previous choice for the next alert rule.

Switching from advanced to default may result in queries and expressions that cannot be converted. In this case, a warning message asks if you want to continue to reset to default settings.

Default and advanced options are enabled by default for Grafana Cloud users and this feature is being rolled out progressively. OSS users can enable them via the alertingQueryAndExpressionsStepMode feature toggle.

Set alert rule name

1. Click Alerts & IRM -> Alert rules -> + New alert rule.

2. Enter a name to identify your alert rule.

   This name is displayed in the alert rule list. It is also the alertname label for every alert instance that is created from this rule.

Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

You can toggle between Default and Advanced options. If the Default vs. Advanced feature is not enabled in your Grafana instance, follow the Advanced options instructions.

Set folder and labels

Organize your alert rule with a folder and set of labels.

In the Labels section, you can optionally choose whether to add labels to organize your alert rules and their notifications. For more details, refer to alert rule labels.

1. Select a folder or click + New folder.

2. Add labels, if required.

   Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

Configure alert evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.

1. Select an evaluation group or click + New evaluation group.

   If you are creating a new evaluation group, specify the interval for the group.

   All rules within the same group are evaluated concurrently over the same time interval.

2. Enter a pending period.

   The pending period is the period in which an alert rule can be in breach of the condition until it fires.

   Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.

3. Turn on pause alert notifications, if required.

   You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and doesn’t create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.

4. In Configure no data and error handling, you can define the alerting behavior and alerting state for two scenarios:

   • When the evaluation returns No data or all values are null.
   • When the evaluation returns Error or timeout.

   Configure no data and error handling

   Configure	Set alert state	Description
   No Data	No Data	The default option for No Data events. Sets alert instance state to No Data. The alert rule also creates a new alert instance DatasourceNoData with the name and UID of the alert rule, and UID of the datasource that returned no data as labels.
   Error	Error	The default option for Error events. Sets alert instance state to Error. The alert rule also creates a new alert instance DatasourceError with the name and UID of the alert rule, and UID of the datasource that returned no data as labels.
   No Data or Error	Alerting	Sets the alert instance state to Pending and then transitions to Alerting once the pending period ends. If you sent the pending period to 0, the alert instance state is immediately set to Alerting.
   No Data or Error	Normal	Sets alert instance state to Normal.
   No Data or Error	Keep Last State	Maintains the alert instance in its last state. Useful for mitigating temporary issues.

   For more details, refer to alert instance states and modify the no data or error state.

Configure notifications

Choose to select a contact point directly from the alert rule form or to use notification policy routing as well as set up mute timings and groupings.

Complete the following steps to set up notifications.

1. Configure who receives a notification when an alert rule fires by either choosing Select contact point or Use notification policy.

   Select contact point

   1. Choose this option to select an existing contact point.

      All notifications for this alert rule are sent to this contact point automatically and notification policies are not used.

   2. You can also optionally select a mute timing as well as groupings and timings to define when not to send notifications.

      Note

      An auto-generated notification policy is generated. Only admins can view these auto-generated policies from the Notification policies list view. Any changes have to be made in the alert rules form.

   Use notification policy

   1. Choose this option to use the notification policy tree to direct your notifications.

      Note

      All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.

   2. Preview your alert instance routing set up.

      Based on the labels added, alert instances are routed to the following notification policies displayed.

   3. Expand each notification policy below to view more details.

   4. Click See details to view alert routing details and an email preview.

Configure notification message

Use annotations to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.

Grafana provides several optional annotations.

1. Optional: Add a summary.

   Short summary of what happened and why.

2. Optional: Add a description.

   Description of what the alert rule does.

3. Optional: Add a Runbook URL.

   Webpage where you keep your runbook for the alert

4. Optional: Add a custom annotation.

   Add any additional information that could help address the alert.

5. Optional: Link dashboard and panel.

   Link the alert rule to a panel to facilitate alert investigation.

6. Click Save rule.