Configure data source-managed alert rules

Data source-managed alert rules can only query Prometheus-based data sources, such as Prometheus, Grafana Mimir, or Grafana Loki. They are one of the two alert rule types supported in Grafana.

Data source-managed alert rules are stored within the data source. In a distributed architecture, they can scale horizontally to provide high-availability.

We recommend using Grafana-managed alert rules whenever possible and opting for data source-managed alert rules when scaling your alerting setup is necessary.

To create or edit data source-managed alert rules, follow these instructions.

Before you begin

Verify that you have write permission to the Prometheus, Mimir, or Loki data source. Otherwise, you cannot create or update data source-managed alert rules.

Enable the Ruler API

For more information, refer to the Mimir Ruler API or Loki Ruler API.

Mimir - use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.
Loki - The local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.

Permissions

Alert rules for Prometheus, Mimir, or Loki instances can be edited or deleted by users with Editor or Admin roles.

If you do not want to manage alert rules for a particular data source, go to its settings and clear the Manage alerts via Alerting UI checkbox.

Provisioning

Note that if you delete an alert resource created in the UI, you can no longer retrieve it.

To backup and manage alert rules, you can provision alerting resources using options such as configuration files, Terraform, or the Alerting API.

Set alert rule name

Click Alerts & IRM -> Alert rules -> + New alert rule.
Enter a name to identify your alert rule.
This name is displayed in the alert rule list. It is also the alertname label for every alert instance that is created from this rule.

Define query and condition

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

Note
By default, new alert rules are Grafana-managed. To switch to Data source-managed, follow these instructions.

Select a Prometheus-based data source from the drop-down list.
You can also click Open advanced data source picker to find more options.
Enter a PromQL or LogQL query, including the alert condition.
In the Rule type option, select Data source-managed.
Click Preview alerts.

Set alert evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

Select a namespace or click + New namespace.
Select an evaluation group or click + New evaluation group.
If you are creating a new evaluation group, specify the interval for the group.
All rules within the same group are evaluated sequentially over the same time interval. You can reorder them from the Alert rules page.
Enter a pending period.
The pending period is the period in which an alert rule can be in breach of the condition until it fires.
Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.

Configure labels and notifications

Add labels to your alert rules to set which notification policy should handle your firing alert instances.

All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy.

Add labels if you want to change the way your notifications are routed.
Add custom labels by selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

Configure notification message

Use annotations to add information to alert messages that can help respond to the alert.

Annotations are included by default in notification messages, and can use text or templates to display dynamic data from queries.

Grafana provides several optional annotations.

Optional: Add a summary.
Short summary of what happened and why.
Optional: Add a description.
Description of what the alert rule does.
Optional: Add a Runbook URL.
Webpage where you keep your runbook for the alert
Optional: Add a custom annotation.
Add any additional information that could help address the alert.
Optional: Link dashboard and panel.
Link the alert rule to a panel to facilitate alert investigation.
Click Save rule.