Introduction to Grafana IRM

Grafana IRM brings on-call management and incident response together in one Grafana Cloud app. It receives alerts from Grafana Alerting and other monitoring tools, and notifies the person on call. Your team declares an incident when the problem needs a coordinated response. Responders investigate in Grafana, beside the data for the alert.

IRM also connects to Service Center. Service Center shows a service’s alerts, incidents, and on-call information in one place. From a service, you can escalate to notify responders.

How IRM works at a glance

An integration in IRM receives an alert from your monitoring tool. A route picks the escalation chain, and IRM adds the alert to a new or open alert group. The escalation chain notifies responders, usually whoever is on call in a schedule, through each person’s notification rules. Acknowledging or resolving the alert group stops escalation. When a problem needs a coordinated response, your team declares an incident.

The following diagram shows the same flow:

---
config:
  fontFamily: "Inter, Helvetica, Arial, sans-serif"
  themeVariables:
    lineColor: "#8b9099"
    fontSize: "14px"
  flowchart:
    nodeSpacing: 28
    rankSpacing: 40
    padding: 14
    subGraphTitleMargin:
      top: 6
      bottom: 10
---
flowchart TB
    accTitle: How Grafana IRM handles an alert
    accDescr {
        An alert from a monitoring tool reaches an integration.
        A route picks the escalation chain, and IRM adds the alert to a new or open alert group.
        The escalation chain runs its steps, the on-call schedule says who is on call,
        and that person's notification rules say how to reach them.
        The responder acknowledges or resolves the alert group, which stops escalation,
        and declares an incident when the problem needs a coordinated response.
    }

    subgraph arrive["1. An alert arrives"]
        direction LR
        alert["Alert
from a monitoring tool"] --> integration["Integration
receives it"]
        integration --> route["Route
picks the escalation chain"]
        route --> group["Alert group
new or open"]
    end

    subgraph notify["2. IRM notifies a responder"]
        direction LR
        chain["Escalation chain
runs its steps"] --> schedule["On-call schedule
says who's on call"]
        schedule --> rules["Notification rules
say how to reach them"]
        rules --> responder["Responder
gets the notification"]
    end

    subgraph respond["3. The team responds"]
        direction LR
        ack["Acknowledge or resolve
stops escalation"]
        incident["Declare an incident
for a coordinated response"]
        ack ~~~ incident
    end

    arrive --> notify --> respond

    classDef step fill:#ffffff,stroke:#F05A28,stroke-width:1px,rx:6,ry:6,color:#1f2937
    classDef stage fill:#f6f7f9,stroke:#e4e6eb,stroke-width:1px,color:#5b6170
    class alert,integration,route,group,chain,schedule,rules,responder,ack,incident step
    class arrive,notify,respond stage

For a closer look at each stage, refer to Introduction to routing and escalation in Grafana IRM.

How you use IRM

Set up a schedule, an escalation chain, and your notification rules so an alert reaches the person on call. Refer to Get started with Grafana IRM.

When IRM notifies you, acknowledge the alert group, investigate, and resolve it. Refer to Respond to alerts in Grafana IRM.

When a problem needs a coordinated response, declare an incident and work through it with your team. Refer to Manage incidents in Grafana IRM.

After you resolve the incident, review what happened and adjust your setup. Refer to Measure and improve incident response in Grafana IRM.

Fundamentals

These are the building blocks of IRM, starting with how IRM receives an alert and how you respond, then incidents, insights, and access.

Integrations

Each alert source, such as Grafana Alerting or another monitoring tool, sends alerts to its own integration.

Integrations also connect chat and collaboration tools, such as Slack and Microsoft Teams, and development and operations tools, such as GitHub, Jira, and Statuspage. Incoming webhooks bring events in from other systems, and outgoing webhooks send IRM events out.

For more information, refer to Connect integrations in Grafana IRM.

Routing and escalation

Routing and escalation decide who is notified about an alert, and when. A route picks which escalation chain runs, based on the alert’s content or labels. Every integration has a default route for alerts that no other route matches. A new integration’s default route has no escalation chain, so nobody is notified until you set one. Related alerts share one alert group. Responders acknowledge or resolve that group.

An escalation chain is a sequence of steps. Steps can notify whoever is on call in a schedule, specific users, or all members of a team. Other steps wait, repeat the chain, declare an incident, or trigger a webhook. Alert templates format alerts and control grouping and auto-resolve. Labels tag alerts so you can route and filter them. Routes, grouping, and templates belong to one integration, and you can reuse an escalation chain across many routes.

For more information, refer to Introduction to routing and escalation in Grafana IRM and Configure alert escalation and routing in Grafana IRM.

On-call schedules

An on-call schedule shows who is on call right now. When an escalation chain notifies a schedule, it reaches whoever holds the current shift. Rotations set the repeating pattern of shifts, and overrides and shift swaps handle changes, such as covering for a teammate who’s away.

For more information, refer to On-call schedules.

Notifications

A responder is anyone IRM notifies. Each person sets their own notification rules, which say how IRM reaches them and in what order. Everyone has a default set and an important set. An escalation step marked as important uses the important set. If a set has no rules, IRM emails the person.

Notification channels are the ways IRM can reach people, such as the Grafana mobile app, SMS, phone calls, email, Slack, and Microsoft Teams.

For more information, refer to Notifications in Grafana IRM and Grafana mobile app.

Alert response

When you acknowledge an alert group, escalation stops. When you resolve the alert group, it closes. Silencing pauses escalation for a set time or indefinitely. When a timed silence ends, escalation starts over from the first step. Responders can also add more users to an alert group. Direct paging notifies people or a team without an alert from a monitoring tool.

For more information, refer to Respond to alerts in Grafana IRM.

Incidents

An alert group is a set of related alerts that you acknowledge or resolve together. An incident is a coordinated response to the problem.

You can declare an incident yourself, from an alert group with Declare incident, or with a Declare incident escalation step on any route except the default route. Each incident has a severity, roles, and a timeline. In Slack, the incident has its own channel. In Microsoft Teams, it’s a conversation thread in an existing channel. A drill is a practice incident. After you resolve an incident, write a post-incident review to help your team learn from it.

For more information, refer to Manage incidents in Grafana IRM.

Insights

Insights help you track your response metrics over time. Alert group insights show how alerts are handled, such as how long responders take to acknowledge an alert group. Incident insights show incident trends, and they don’t include drills.

For more information, refer to Measure and improve incident response in Grafana IRM.

Access and teams

User roles control what each person can do in IRM. They include the basic Grafana roles, such as Viewer, Editor, and Admin, plus IRM role-based access control (RBAC) roles, such as OnCaller. Teams group people with the resources they own, such as integrations, schedules, and escalation chains, and control who can see those resources.

For more information, refer to Manage access for Grafana IRM.