--- title: "Connect Grafana Alerting and Grafana IRM | Grafana Cloud documentation" description: "Connect Grafana Alerting to Grafana IRM, route alerts, choose who gets paged, and troubleshoot the connection." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Connect Grafana Alerting and Grafana IRM Connect Grafana Alerting to Grafana IRM so your alert rules page the right on-call responders. This page covers setup, routing, managing the connection, and troubleshooting. To learn how the two products work together, refer to [Grafana Alerting integration for Grafana IRM](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting). If you run Grafana OSS or Grafana Enterprise, refer to [Connect a self-managed Grafana instance](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#connect-a-self-managed-grafana-instance). ## Before you begin Before you connect Grafana Alerting to IRM, make sure you have the following: - The IRM **Admin** or **Integrations Editor** role to create the integration and choose who gets paged. If you use Quick connect, IRM creates the contact point for you, so you don’t need Alerting permissions for that step. - The **Editor** or **Admin** role in Grafana to create notification policies, and to create contact points if you start in Alerting. For details, refer to [Roles and permissions for Grafana IRM](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/set-up/manage-access/roles-and-permissions). - Alert rules that use the built-in Grafana Alertmanager. The **Grafana IRM** contact point is only available there. - An escalation chain and the on-call schedule or users it notifies. To create one, refer to [Configure escalation chains](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/escalation-and-routing/escalation-chains). ## Create the integration and contact point Create a Grafana Alerting integration in IRM and a **Grafana IRM** contact point in Alerting. You can start in either product. From IRM (Quick connect)From Grafana Alerting (contact point) From IRM (Quick connect) From Grafana Alerting (contact point) 1. Navigate to **Alerts & IRM** > **IRM** > **Integrations** > **Monitoring Systems**. 2. Click **+ Add integration**. 3. On the **Grafana Alerting** tile, click **Quick connect**. 4. Enter a name and choose the team that owns the integration. 5. Choose a contact point: - **Connect existing Contact point** (the default) adds IRM to a contact point you already have. - **Create a new one** creates a new contact point. - Leave the contact point empty to create only the integration. 6. Save the integration. If IRM can’t set up the contact point, it still creates the integration and shows a message. You can connect a contact point later. Refer to [Manage the connection](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#manage-the-connection). 1. Navigate to **Alerts & IRM** > **Alerting** > **Notification configuration** > **Contact points**. 2. Click **+ New contact point** and enter a name. 3. From the **Integration** list, select **Grafana IRM**. 4. Choose **New IRM integration** and enter a name, or choose **Existing IRM integration** and select it. 5. Under the optional notification settings, leave **Disable resolved message** turned off, so IRM alert groups resolve automatically. Refer to [Resolving](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#resolving). 6. Click **Save contact point**. ## Route alerts to the contact point IRM starts receiving alerts when a notification policy routes them to the **Grafana IRM** contact point. 1. Navigate to **Alerts & IRM** > **Alerting** > **Notification configuration** > **Notification policies**. 2. Click **+ New notification policy**, or select an existing policy to edit. 3. Match the alerts you want to page on, for example, `severity=critical`. 4. Select the **Grafana IRM** contact point. 5. Choose the **Group by** labels. They control how alerts are grouped in IRM. Refer to [Grouping](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#grouping). 6. Save the policy. For details, refer to [Configure notification policies](https://grafana.com/docs/grafana-cloud/observe-and-act/alert-and-measure-reliability/alerting/configure-notifications/create-notification-policy/). You can also select the contact point directly in an alert rule. Refer to [Configure notifications](https://grafana.com/docs/grafana-cloud/observe-and-act/alert-and-measure-reliability/alerting/alerting-rules/create-grafana-managed-rule/#configure-notifications). When a policy uses the contact point, the IRM integration page stops showing “Check the notification policy for this contact point in Grafana Alerting.” ## Choose who gets paged A new IRM integration starts with a default route that has no escalation chain. Attach one so IRM pages your responders. 1. In IRM, open the Grafana Alerting integration. 2. On the default route, set **Trigger escalation chain** to the escalation chain that pages your responders. To send different alerts to different teams, add routes above the default route. Refer to [Route alerts by team and severity](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#route-alerts-by-team-and-severity). ## Send a test alert Send a test alert from the contact point to check the connection. 1. In Alerting, open the contact point and click **Test**. 2. In IRM, check that a new alert group appears and that the escalation chain pages your responders. 3. Resolve the test alert group in IRM. Test alerts never resolve on their own, so each test leaves an open alert group until you resolve it. To learn what happens to real alert groups after setup, refer to [The life of an alert group](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#the-life-of-an-alert-group). ## Secure the integration with a bearer token You can require every notification sent to the IRM integration to include a Grafana service account token. Without this, IRM accepts any request sent to the integration URL. To set this up, you need: - A Grafana [external service account](https://grafana.com/docs/grafana/latest/administration/service-accounts/) with the **alert groups: write** permission (`grafana-irm-app.alert-groups:write`). - A service account token for that account. It starts with `glsa_`. > Caution > > Add the token to the contact point before you turn on the requirement in IRM. Otherwise, IRM rejects notifications sent between the two steps with `403 Forbidden`, and you lose them. 1. Navigate to **Alerts & IRM** > **Alerting** > **Notification configuration** > **Contact points**. 2. Edit the **Grafana IRM** contact point connected to the integration. 3. Under **Optional Grafana IRM settings**, set **Authentication Header - Scheme** to `Bearer` and **Authentication Header - Credentials** to the token, then save. Leave the Basic authentication fields empty, because the contact point accepts only one kind of authentication. 4. In IRM, open the integration, click the **More options** menu, and select **Integration Settings**. 5. Turn on **Require a Grafana service account token to send alerts to this integration** and click **Update Integration**. The **Require a Grafana service account token** switch only appears when you edit an integration, not when you create one. If notifications stop arriving after you turn this on, check the response IRM returns: - `403 Forbidden`: the token is missing, malformed, revoked, expired, or lacks the permission. - `503 Service Unavailable` or `429 Too Many Requests`: IRM couldn’t check the token with Grafana. ## Route alerts by team and severity Use notification policies in Alerting to choose which alerts reach IRM, and IRM routes to choose which escalation chain runs. To learn how the two stages fit together, refer to [How routing works across both products](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#how-routing-works-across-both-products). For route types and how to manage them, refer to [Configure routing rules](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/escalation-and-routing/routing-rules). ### Write route templates Start by choosing the labels that identify ownership and severity, such as `team` and `severity`. Routes work best on labels that have the same value for every alert in a [notification group](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#grouping). Route templates read these shared labels from `payload.commonLabels`. To make sure a routing label is always shared, add it to **Group by**. For a notification with these common labels: JSON ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```json { "commonLabels": { "alertname": "HighLatency", "service": "checkout", "severity": "critical" } } ``` The following route template evaluates to `True`, so the route matches: jinja2 ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```jinja2 {{ payload.commonLabels.service == "checkout" and payload.commonLabels.severity == "critical" }} ``` Use `payload.commonLabels`, not `payload.labels`. Notifications from Alerting don’t have a top-level `labels` field. In a template, `labels` refers to the alert group’s IRM labels. IRM checks every notification against the routes. If a later notification matches a different route, IRM starts a new alert group on that route. ### Copy Alerting labels into IRM labels IRM adds only a `service_name` label automatically. To filter alert groups or build label-based routes on other Alerting labels, copy them into IRM labels with a multi-label extraction template: 1. On the IRM integration page, click the edit icon next to **Labels schema**. 2. Open the **Multi-label extraction template** and enter a template. For example, to copy every common label: jinja2 ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```jinja2 {{ payload.commonLabels | tojson }} ``` 3. Save the template, then click **Update Integration**. For the notification in the previous section, the alert group gets these labels, plus `service_name`: - `alertname: HighLatency` - `service: checkout` - `severity: critical` IRM sets labels when it creates an alert group, up to 50 labels. Later notifications don’t change them. For static and dynamic labels, refer to [Configure labels](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/escalation-and-routing/labels/configure-labels). A `team` label in Alerting doesn’t set the IRM team. The IRM team that owns an integration comes from Grafana teams, and `team_name` is the default label key IRM uses for teams. Refer to [Manage teams in Grafana IRM](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/set-up/manage-access/teams). ## Manage the connection After setup, you can manage connected contact points, stay within rate limits, and find options for heartbeat monitoring. ### Connected contact points To see the contact points that send to an IRM integration, open the integration and click the edit icon next to **Contact point**. The **Connected Contact Points** list flags any contact point that no notification policy uses. From there, you can connect an existing contact point, create a new one, or disconnect one. When you delete the IRM integration, IRM disconnects it from each contact point but keeps the contact points. Check Alerting for contact points that no longer send anywhere. ### Limits IRM accepts up to 300 notifications per 5 minutes per integration, and 900 per 5 minutes per organization. Notifications over the limit get `429 Too Many Requests`. To send fewer notifications, group more alerts with **Group by** or use a longer group interval. The contact point’s **Max Alerts** setting limits how many alerts each notification includes. `commonLabels` reflects only the alerts included in the notification. The `truncatedAlerts` field reports how many alerts were left out. ### Heartbeat monitoring To get notified if Grafana Alerting stops sending notifications, set up heartbeat monitoring through a separate Webhook integration: 1. In IRM, create a Webhook integration, turn on heartbeat monitoring, and copy the heartbeat URL. Refer to [Use a Webhook integration as a heartbeat receiver](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/configure-integrations/#use-a-webhook-integration-as-a-heartbeat-receiver). 2. In Alerting, create a **Webhook** contact point that sends to the heartbeat URL. 3. Create an alert rule that always fires, for example, with the query `vector(1) > 0`, and select the Webhook contact point directly in the rule. Set the rule’s timing so it sends before the heartbeat interval in IRM elapses: - **Pending period** and **Repeat interval**: shorter than the heartbeat interval. - **Group wait**: `0s`. - **Group interval**: `1s`. For more details on the Alerting side, refer to [Enable heartbeat monitoring in Grafana IRM](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/configure-irm/#enable-heartbeat-monitoring-in-grafana-irm-optional). ## Connect a self-managed Grafana instance If you run Grafana OSS or Grafana Enterprise and use Grafana Cloud IRM, connect them with a Webhook contact point and an IRM Alertmanager integration. 1. In IRM, create an [Alertmanager integration](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/alert-sources/alertmanager) and copy its URL. 2. In your Grafana instance, create a **Webhook** contact point with that URL, and route alerts to it with a notification policy. For the Alerting steps, refer to [Configure Grafana IRM for Alerting](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/integrations/configure-irm/) in the Grafana documentation. The Alertmanager integration groups, resolves, and routes alerts the same way as the Grafana Alerting integration, as described in [Grafana Alerting integration for Grafana IRM](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting). Quick connect, contact point sync, and the automatic `service_name` label are part of the Grafana Cloud integration. The Alertmanager integration also supports [heartbeat monitoring](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/configure-integrations/#enable-heartbeat-monitoring). ## Troubleshoot Find your symptom to see the likely causes and how to fix them. ### No alerts arrive in IRM - **No notification policy uses the contact point.** Add one. Refer to [Route alerts to the contact point](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#route-alerts-to-the-contact-point). - **IRM requires a token that the contact point doesn’t send.** IRM rejects these notifications with `403 Forbidden`. Add the token to the contact point. Refer to [Secure the integration with a bearer token](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#secure-the-integration-with-a-bearer-token). - **Grafana IRM is unavailable in the contact point form,** and shows **Enable IRM through a Webhook integration**. IRM isn’t available in your stack. Contact your Grafana Cloud administrator. ### Notifications get `429 Too Many Requests` Notifications exceed the IRM rate limit. Refer to [Limits](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#limits). If you require a bearer token, IRM also returns `429` when it can’t check the token with Grafana. ### Alert groups appear, but no one is paged The matching route has no escalation chain. Refer to [Choose who gets paged](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/connect/#choose-who-gets-paged). ### An alert group doesn’t resolve For how resolving works, refer to [Resolving](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#resolving). - **Some alerts in the notification group are still firing.** The alert group resolves when every alert clears. - **Disable resolved message is turned on.** Turn it off on the contact point, then resolve the alert group in IRM. - **The alert resolved while it was silenced or muted in Alerting.** Resolve the alert group in IRM. Refer to [Silences and maintenance](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#silences-and-maintenance). - **Someone edited the Grouping Id template.** Restore the default, `{{ payload.groupKey }}`. - **The notification policy or Group by changed while alerts were firing.** New notifications start a new alert group, and the old one stays open. Resolve the old alert group in IRM. Refer to [Grouping](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#grouping). - **The resolved notification matched a different route than the firing one.** The original alert group stays open. Resolve it in IRM, and check that your route templates match both firing and resolved notifications. - **It’s a test alert group.** Test alerts don’t resolve on their own. Resolve it in IRM. ### Too many or too few alert groups **Group by** and the notification policy path control grouping. Change **Group by** to fit how you want alerts grouped. New alert groups after you change a policy or **Group by** are expected. Refer to [Grouping](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/grafana-alerting/#grouping). ### A route doesn’t match - **The template uses `payload.labels`.** Use `payload.commonLabels.