Documentation Grafana Cloud Observe and act Manage on-call and incidents Connect integrations Grafana Alerting

Grafana Alerting integration for Grafana IRM

This page explains the concepts behind connecting Grafana Alerting to Grafana IRM in Grafana Cloud: how alerts move between the two products, and how grouping, resolving, silences, and routing behave. For setup and configuration steps, refer to Connect Grafana Alerting and Grafana IRM.

How Grafana Alerting works with Grafana IRM

Grafana Alerting and Grafana IRM work together to page on-call responders when your alert rules fire. Alerting decides when to send notifications, how to group alerts, and when they resolve. IRM decides who gets paged and how.

---
config:
  fontFamily: "Inter, Helvetica, Arial, sans-serif"
  themeVariables:
    lineColor: "#8b9099"
    fontSize: "14px"
  flowchart:
    nodeSpacing: 28
    rankSpacing: 40
    padding: 14
    subGraphTitleMargin:
      top: 6
      bottom: 10
---
flowchart LR
    accTitle: How an alert moves from Grafana Alerting to Grafana IRM
    accDescr {
        In Grafana Alerting, an alert rule fires, a notification policy matches its labels and applies Group by,
        and the Grafana IRM contact point sends one notification per group.
        In Grafana IRM, the Grafana Alerting integration receives it, the first matching route picks the alert group,
        and the escalation chain pages responders.
        You configure the notification policy and the escalation chain yourself.
    }

    subgraph alerting["Grafana Alerting"]
        direction TB
        rule["`**Alert rule**
        fires alerts`"] --> policy["`**Notification policy**
        matches labels, applies Group by`"]
        policy --> cp["`**Grafana IRM contact point**
        one notification per group`"]
    end

    subgraph irm["Grafana IRM"]
        direction TB
        int["`**Grafana Alerting integration**
        receives the notification`"] --> route["`**Route**
        first match wins`"]
        route --> ag["`**Alert group**
        new or open`"]
        ag --> ec["`**Escalation chain**
        pages responders`"]
    end

    alerting --> irm

    classDef step fill:#ffffff,stroke:#F05A28,stroke-width:1px,rx:6,ry:6,color:#1f2937
    classDef configure fill:#ffffff,stroke:#F05A28,stroke-width:2px,stroke-dasharray:5 4,rx:6,ry:6,color:#1f2937
    classDef stage fill:#f6f7f9,stroke:#e4e6eb,stroke-width:1px,color:#5b6170
    class rule,cp,int,route,ag step
    class policy,ec configure
    class alerting,irm stage

You configure the two dashed steps yourself: a notification policy in Alerting and an escalation chain in IRM. The contact point sends one notification for each Alerting notification group, not for each alert.

The life of an alert group

Here’s what happens to one alert group, from the first firing notification to the resolve:

---
config:
  fontFamily: "Inter, Helvetica, Arial, sans-serif"
  themeVariables:
    fontSize: "14px"
    lineColor: "#8b9099"
    actorBkg: "#ffffff"
    actorBorder: "#F05A28"
    actorTextColor: "#1f2937"
    actorLineColor: "#d0d3d9"
    signalColor: "#8b9099"
    signalTextColor: "#1f2937"
    sequenceNumberColor: "#ffffff"
  sequence:
    mirrorActors: true
---
sequenceDiagram
    accTitle: The life of an alert group
    accDescr {
        Grafana Alerting sends a firing notification for a new group, and IRM creates an alert group and pages the responder.
        Repeat notifications for the same group join the open alert group. The responder acknowledges in IRM.
        When every alert in the group clears, Alerting sends a resolved notification and the alert group resolves.
    }
    autonumber
    box rgb(246, 247, 249)
        participant A as Grafana Alerting
        participant I as Grafana IRM
        participant R as Responder
    end
    A->>I: Firing, new group
    I->>R: New alert group, responder paged
    A->>I: Repeat, same group
    R->>I: Acknowledge
    A->>I: Resolved, every alert cleared
    I-->>R: Alert group resolves

IRM adds repeat notifications to the open alert group without restarting the escalation. Two cases work differently:

Responders acknowledge alert groups in IRM. Acknowledging or resolving an alert group in IRM doesn’t change the alert rule in Alerting.

Grouping

The Group by setting in Alerting controls how alerts are grouped in IRM. Notifications from the same Alerting notification group join the same open IRM alert group. Alerts that match different IRM routes never share an alert group.

Repeat notifications don’t restart the escalation, but they do count toward escalation steps that check how many alerts arrive in a time window.

For example, suppose a HighLatency rule fires three alerts:

Here’s how many IRM alert groups you get with two Group by settings:

To get one alert group per team or service, add that label to Group by.

These changes in Alerting start new IRM alert groups for new alerts:

• Changing the notification policy tree or its matchers
• Changing Group by
• Renaming a contact point that’s selected directly in an alert rule

If you make one of these changes while alerts are firing, the old alert group stays open. Resolve it in IRM.

For more information, refer to Group alert notifications and Configure alert grouping.

Resolving

An IRM alert group resolves automatically when every alert in its Alerting notification group has resolved. Until then, it stays firing. After it resolves, the next firing notification starts a new alert group.

The resolved notification must match the same IRM route as the firing notifications. If it matches a different route, the original alert group stays open.

Alerting also resolves alerts whose series stop returning data, after two evaluations by default. IRM receives these as resolved notifications. Refer to Stale alert instances.

To resolve alert groups automatically when an escalation finishes, add the Resolve alert group automatically step to the escalation chain.

Silences and maintenance

Alerting and IRM each manage their own silences, so you can quiet alerts at the level that fits the situation. An IRM silence pauses paging for one alert group and leaves Alerting as it is. If the alert resolves during an IRM silence, IRM resolves the alert group when the silence ends. An Alerting silence stops notifications at the source and leaves existing IRM alert groups as they are.

Silencing one alert in a notification group doesn’t resolve it in IRM. Later notifications leave that alert out.

IRM maintenance mode offers Silence escalations, Group alerts, and Disable alerts. Refer to Manage maintenance mode, Configure silences, and Configure mute timings and active time intervals.

How routing works across both products

Routing happens in two stages. Notification policies in Alerting choose which alerts reach IRM, for example, only critical and warning alerts. IRM routes then choose which escalation chain runs, for example, by team.

IRM checks routes from top to bottom and uses the first match. The default route comes last and catches everything else. Unlike notification policies, IRM routes don’t nest. To write route templates for Alerting notifications, refer to Route alerts by team and severity.

Next steps

• Connect Grafana Alerting and Grafana IRM
• Configure alert grouping
• Configure routing rules
• Group alert notifications