Alert rule evaluation

This is documentation for the next version of Grafana documentation. For the latest stable release, go to the latest version.

Grafana Cloud Enterprise Open source RSS

Alert rule evaluation

The criteria determining when an alert rule fires are based on three settings:

These settings affect how alert instances progress through their lifecycle.

Alerting lifecycle

Each alert rule can generate one or more alert instances.

An alert instance transitions between these common states based on how long the alert condition remains met or not met.

StateDescription
NormalThe state of an alert when the condition (threshold) is not met.
PendingThe state of an alert that has breached the threshold but for less than the pending period.
AlertingThe state of an alert that has breached the threshold for longer than the pending period.
RecoveringThe state of a firing alert when the threshold is no longer breached, but for less than the keep firing for period.
A diagram of the lifecyle of a firing alert instance.

If an alert rule changes (except for updates to annotations, the evaluation interval, or other internal fields), its alert instances reset to the Normal state, and update accordingly during the next evaluation.

Note

To learn about additional alert instance states, see No Data and Error states.

Notification routing

Alert instances are routed for notifications in two scenarios:

  1. When they transition to the Alerting state.
  2. When they transition to Normal state and marked as Resolved, either from the Alerting or Recovering state.

Evaluation group

Every alert rule and recording rule is assigned to an evaluation group.

Each evaluation group contains an evaluation interval that determines how frequently the rule is checked. For instance, the evaluation may occur every 10s, 30s, 1m, 10m, etc.

Rules can be evaluated concurrently or sequentially. For details, see How rules are evaluated within a group.

Pending period

You can set a Pending period to prevent unnecessary notifications caused by temporary issues.

When the alert condition is met, the alert instance enters the Pending state. It remains in this state until the condition has been continuously true for the entire Pending period.

This ensures the condition breach is stable before the alert transitions to the Alerting state and routed for notification.

  • Normal -> Pending -> Alerting*

You can also set the Pending period to zero to skip the Pending state entirely and transition to Alerting immediately.

Keep firing for

You can set a Keep firing for period to avoid repeated firing-resolving-firing notifications caused by flapping conditions.

When the alert condition is no longer met during the Alerting state, the alert instance enters the Recovering state.

  • AlertingRecoveringNormal (Resolved)*
  • After the Keep firing for period elapses, the alert transitions to the Normal state and is marked as Resolved.
  • If the alert condition is met again, the alert transitions back to the Alerting state, and no new notifications are sent.

You can also set the Keep firing for period to zero to skip the Recovering state entirely.

Evaluation example

Keep in mind:

  • One alert rule can generate multiple alert instances—one for each series or dimension produced by the rule’s query. Alert instances from the same alert rule may be in different states.
  • Only alert instances in the Alerting and Normal (Resolved) state are routed for notifications.

Consider an alert rule with an evaluation interval set at every 30 seconds and a pending period of 90 seconds. The evaluation occurs as follows:

TimeConditionAlert instance statePending counter
00:30 (first evaluation)Not metNormal-
01:00 (second evaluation)BreachedPending0s
01:30 (third evaluation)BreachedPending30s
02:00 (fourth evaluation)BreachedPending60s
02:30 (fifth evaluation)BreachedAlerting 📩90s

With a keep firing for period of 0 seconds, the alert instance transitions immediately from Alerting to Normal, and marked as Resolved:

TimeConditionAlert instance statePending counter
03:00 (sixth evaluation)Not metNormal Resolved 📩120s
03:30 (seventh evaluation)Not metNormal150s