Grafana Cloud Enterprise Open source RSS

Alert rule evaluation

The criteria determining when an alert rule fires are based on two settings:

Set the evaluation behavior of the alert rule in Grafana.
Set alert rule evaluation

Evaluation group

Every alert rule is assigned to an evaluation group. You can assign the alert rule to an existing evaluation group or create a new one.

Each evaluation group contains an evaluation interval that determines how frequently the alert rule is checked. For instance, the evaluation may occur every 10s, 30s, 1m, 10m, etc.

Evaluation strategies

Alert rules in different groups can be evaluated simultaneously.

  • Grafana-managed alert rules within the same group are evaluated concurrently—they are evaluated at different times over the same evaluation interval but display the same evaluation timestamp.

  • Data-source managed alert rules within the same group are evaluated sequentially, one after the other—this is necessary to ensure that recording rules are evaluated before alert rules.

Pending period

You can set a pending period to prevent unnecessary alerts from temporary issues.

The pending period specifies how long the condition must be met before firing, ensuring the condition is consistently met over a consecutive period.

You can also set the pending period to zero to skip it and have the alert fire immediately once the condition is met.

Evaluation example

Keep in mind:

  • One alert rule can generate multiple alert instances - one for each time series produced by the alert rule’s query.
  • Alert instances from the same alert rule may be in different states. For instance, only one observed machine might start firing.
  • Only Alerting and Resolved alert instances are routed to manage their notifications.
A diagram of the alert instance states and when to route their notifications.

Consider an alert rule with an evaluation interval set at every 30 seconds and a pending period of 90 seconds. The evaluation occurs as follows:

TimeConditionAlert instance statePending counter
00:30 (first evaluation)Not metNormal-
01:00 (second evaluation)BreachedPending0s
01:30 (third evaluation)BreachedPending30s
02:00 (fourth evaluation)BreachedPending60s
02:30 (fifth evaluation)BreachedAlerting*90s

An alert instance is resolved when it transitions from the Firing to the Normal state. For instance, in the previous example:

TimeConditionAlert instance statePending counter
03:00 (sixth evaluation)Not metNormal Resolved *120s
03:30 (seventh evaluation)Not metNormal150s

To learn more about the state changes of alert rules and alert instances, refer to State and health of alert rules.