Documentation Grafana Cloud Observe and act Manage on-call and incidents Connect integrations Grafana Alerting Connect Alerting and IRM

Connect Grafana Alerting and Grafana IRM

Connect Grafana Alerting to Grafana IRM so your alert rules page the right on-call responders. This page covers setup, routing, managing the connection, and troubleshooting. To learn how the two products work together, refer to Grafana Alerting integration for Grafana IRM.

If you run Grafana OSS or Grafana Enterprise, refer to Connect a self-managed Grafana instance.

Before you begin

Before you connect Grafana Alerting to IRM, make sure you have the following:

• The IRM Admin or Integrations Editor role to create the integration and choose who gets paged. If you use Quick connect, IRM creates the contact point for you, so you don’t need Alerting permissions for that step.
• The Editor or Admin role in Grafana to create notification policies, and to create contact points if you start in Alerting. For details, refer to Roles and permissions for Grafana IRM.
• Alert rules that use the built-in Grafana Alertmanager. The Grafana IRM contact point is only available there.
• An escalation chain and the on-call schedule or users it notifies. To create one, refer to Configure escalation chains.

Create the integration and contact point

Create a Grafana Alerting integration in IRM and a Grafana IRM contact point in Alerting. You can start in either product.

Route alerts to the contact point

IRM starts receiving alerts when a notification policy routes them to the Grafana IRM contact point.

1. Navigate to Alerts & IRM > Alerting > Notification configuration > Notification policies.
2. Click + New notification policy, or select an existing policy to edit.
3. Match the alerts you want to page on, for example, severity=critical.
4. Select the Grafana IRM contact point.
5. Choose the Group by labels. They control how alerts are grouped in IRM. Refer to Grouping.
6. Save the policy.

For details, refer to Configure notification policies. You can also select the contact point directly in an alert rule. Refer to Configure notifications.

When a policy uses the contact point, the IRM integration page stops showing “Check the notification policy for this contact point in Grafana Alerting.”

Choose who gets paged

A new IRM integration starts with a default route that has no escalation chain. Attach one so IRM pages your responders.

1. In IRM, open the Grafana Alerting integration.
2. On the default route, set Trigger escalation chain to the escalation chain that pages your responders.

To send different alerts to different teams, add routes above the default route. Refer to Route alerts by team and severity.

Send a test alert

Send a test alert from the contact point to check the connection.

1. In Alerting, open the contact point and click Test.
2. In IRM, check that a new alert group appears and that the escalation chain pages your responders.
3. Resolve the test alert group in IRM.

Test alerts never resolve on their own, so each test leaves an open alert group until you resolve it. To learn what happens to real alert groups after setup, refer to The life of an alert group.

Secure the integration with a bearer token

You can require every notification sent to the IRM integration to include a Grafana service account token. Without this, IRM accepts any request sent to the integration URL.

To set this up, you need:

• A Grafana external service account with the alert groups: write permission (grafana-irm-app.alert-groups:write).
• A service account token for that account. It starts with glsa_.

1. Navigate to Alerts & IRM > Alerting > Notification configuration > Contact points.
2. Edit the Grafana IRM contact point connected to the integration.
3. Under Optional Grafana IRM settings, set Authentication Header - Scheme to Bearer and Authentication Header - Credentials to the token, then save. Leave the Basic authentication fields empty, because the contact point accepts only one kind of authentication.
4. In IRM, open the integration, click the More options menu, and select Integration Settings.
5. Turn on Require a Grafana service account token to send alerts to this integration and click Update Integration.

The Require a Grafana service account token switch only appears when you edit an integration, not when you create one.

If notifications stop arriving after you turn this on, check the response IRM returns:

• 403 Forbidden: the token is missing, malformed, revoked, expired, or lacks the permission.
• 503 Service Unavailable or 429 Too Many Requests: IRM couldn’t check the token with Grafana.

Route alerts by team and severity

Use notification policies in Alerting to choose which alerts reach IRM, and IRM routes to choose which escalation chain runs. To learn how the two stages fit together, refer to How routing works across both products. For route types and how to manage them, refer to Configure routing rules.

Write route templates

Start by choosing the labels that identify ownership and severity, such as team and severity. Routes work best on labels that have the same value for every alert in a notification group. Route templates read these shared labels from payload.commonLabels. To make sure a routing label is always shared, add it to Group by.

For a notification with these common labels:

JSON Copy

{
  "commonLabels": {
    "alertname": "HighLatency",
    "service": "checkout",
    "severity": "critical"
  }
}

The following route template evaluates to True, so the route matches:

jinja2 Copy

{{ payload.commonLabels.service == "checkout" and payload.commonLabels.severity == "critical" }}

Use payload.commonLabels, not payload.labels. Notifications from Alerting don’t have a top-level labels field. In a template, labels refers to the alert group’s IRM labels.

IRM checks every notification against the routes. If a later notification matches a different route, IRM starts a new alert group on that route.

Copy Alerting labels into IRM labels

IRM adds only a service_name label automatically. To filter alert groups or build label-based routes on other Alerting labels, copy them into IRM labels with a multi-label extraction template:

1. On the IRM integration page, click the edit icon next to Labels schema.

2. Open the Multi-label extraction template and enter a template. For example, to copy every common label:

   jinja2 Copy

   {{ payload.commonLabels | tojson }}

3. Save the template, then click Update Integration.

For the notification in the previous section, the alert group gets these labels, plus service_name:

• alertname: HighLatency
• service: checkout
• severity: critical

IRM sets labels when it creates an alert group, up to 50 labels. Later notifications don’t change them. For static and dynamic labels, refer to Configure labels.

A team label in Alerting doesn’t set the IRM team. The IRM team that owns an integration comes from Grafana teams, and team_name is the default label key IRM uses for teams. Refer to Manage teams in Grafana IRM.

Manage the connection

After setup, you can manage connected contact points, stay within rate limits, and find options for heartbeat monitoring.

Connected contact points

To see the contact points that send to an IRM integration, open the integration and click the edit icon next to Contact point. The Connected Contact Points list flags any contact point that no notification policy uses. From there, you can connect an existing contact point, create a new one, or disconnect one.

When you delete the IRM integration, IRM disconnects it from each contact point but keeps the contact points. Check Alerting for contact points that no longer send anywhere.

Limits

IRM accepts up to 300 notifications per 5 minutes per integration, and 900 per 5 minutes per organization. Notifications over the limit get 429 Too Many Requests. To send fewer notifications, group more alerts with Group by or use a longer group interval.

The contact point’s Max Alerts setting limits how many alerts each notification includes. commonLabels reflects only the alerts included in the notification. The truncatedAlerts field reports how many alerts were left out.

Heartbeat monitoring

To get notified if Grafana Alerting stops sending notifications, set up heartbeat monitoring through a separate Webhook integration:

1. In IRM, create a Webhook integration, turn on heartbeat monitoring, and copy the heartbeat URL. Refer to Use a Webhook integration as a heartbeat receiver.
2. In Alerting, create a Webhook contact point that sends to the heartbeat URL.
3. Create an alert rule that always fires, for example, with the query vector(1) > 0, and select the Webhook contact point directly in the rule.

Set the rule’s timing so it sends before the heartbeat interval in IRM elapses:

• Pending period and Repeat interval: shorter than the heartbeat interval.
• Group wait: 0s.
• Group interval: 1s.

For more details on the Alerting side, refer to Enable heartbeat monitoring in Grafana IRM.

Connect a self-managed Grafana instance

If you run Grafana OSS or Grafana Enterprise and use Grafana Cloud IRM, connect them with a Webhook contact point and an IRM Alertmanager integration.

1. In IRM, create an Alertmanager integration and copy its URL.
2. In your Grafana instance, create a Webhook contact point with that URL, and route alerts to it with a notification policy.

For the Alerting steps, refer to Configure Grafana IRM for Alerting in the Grafana documentation.

The Alertmanager integration groups, resolves, and routes alerts the same way as the Grafana Alerting integration, as described in Grafana Alerting integration for Grafana IRM. Quick connect, contact point sync, and the automatic service_name label are part of the Grafana Cloud integration. The Alertmanager integration also supports heartbeat monitoring.

Troubleshoot

Find your symptom to see the likely causes and how to fix them.

No alerts arrive in IRM

• No notification policy uses the contact point. Add one. Refer to Route alerts to the contact point.
• IRM requires a token that the contact point doesn’t send. IRM rejects these notifications with 403 Forbidden. Add the token to the contact point. Refer to Secure the integration with a bearer token.
• Grafana IRM is unavailable in the contact point form, and shows Enable IRM through a Webhook integration. IRM isn’t available in your stack. Contact your Grafana Cloud administrator.

Notifications get 429 Too Many Requests

Notifications exceed the IRM rate limit. Refer to Limits. If you require a bearer token, IRM also returns 429 when it can’t check the token with Grafana.

Alert groups appear, but no one is paged

The matching route has no escalation chain. Refer to Choose who gets paged.

An alert group doesn’t resolve

For how resolving works, refer to Resolving.

• Some alerts in the notification group are still firing. The alert group resolves when every alert clears.
• Disable resolved message is turned on. Turn it off on the contact point, then resolve the alert group in IRM.
• The alert resolved while it was silenced or muted in Alerting. Resolve the alert group in IRM. Refer to Silences and maintenance.
• Someone edited the Grouping Id template. Restore the default, {{ payload.groupKey }}.
• The notification policy or Group by changed while alerts were firing. New notifications start a new alert group, and the old one stays open. Resolve the old alert group in IRM. Refer to Grouping.
• The resolved notification matched a different route than the firing one. The original alert group stays open. Resolve it in IRM, and check that your route templates match both firing and resolved notifications.
• It’s a test alert group. Test alerts don’t resolve on their own. Resolve it in IRM.

Too many or too few alert groups

Group by and the notification policy path control grouping. Change Group by to fit how you want alerts grouped. New alert groups after you change a policy or Group by are expected. Refer to Grouping.

A route doesn’t match

• The template uses payload.labels. Use payload.commonLabels.<label> instead.
• The label isn’t shared across the notification group. Add the label to Group by so every alert in the group has the same value.

Next steps

• Grafana Alerting integration for Grafana IRM
• Configure routing rules
• Configure alert grouping
• Configure labels
• Configure notification policies
• Group alert notifications