Documentation Grafana Cloud Alerts and IRM IRM Connect integrations Alert sources Grafana Alerting

Grafana Alerting integration for Grafana IRM

Grafana Alerting for Grafana IRM can be set up using two methods:

• Grafana IRM is connected to the same Grafana instance being used to manage Grafana IRM.
• Grafana IRM is connected to one or more Grafana instances, separate from the one being used to manage Grafana IRM.

Configure Grafana Alerting in same Grafana instance

Use the following method if you are connecting Grafana IRM with alerts coming from the same Grafana instance from which Grafana IRM is being managed.

1. In Grafana IRM, navigate to IRM > Integrations > Monitoring Systems

2. Click Quick connect in the Grafana Alerting tile. This will open a New Grafana Alerting integration configuration window

3. In the opened window, enter a name and description for the integration and choose existing or create a new contact point. This contact point will send alerts to the created integration

   Note

   You must connect the contact point with a notification policy in Grafana Alerting to receive alerts in Grafana IRM. For more information, see Contact points in Grafana Alerting

4. Determine the escalation chain for the new integration by either selecting an existing one or by creating a new escalation chain

5. In Grafana Cloud Alerting, navigate to Alerting > Contact Points and find a contact point with a name matching the integration you created in Grafana IRM

6. Click the Edit (pencil) icon, then click Test. This will send a test alert to Grafana IRM

Configure external Grafana Alerting from other Grafana Instance

Connect Grafana IRM with alerts coming from a Grafana instance that is different from the instance that Grafana IRM is being managed:

1. In Grafana IRM, navigate to IRM > Integrations > Monitoring Systems

2. Select the Alertmanager tile

3. Enter a name and description for the integration, click Create

4. A new page will open with the integration details. Copy the IRM Integration URL from HTTP Endpoint section

5. Go to the other Grafana instance to connect to Grafana IRM and navigate to Alerting > Contact Points

6. Select New Contact Point

7. Choose the contact point type webhook, then paste the URL generated in step 3 into the URL field

   Note

   You must connect the contact point with a notification policy in Grafana Alerting to receive alerts in Grafana IRM. For more information, see Contact points in Grafana Alerting

8. Click the Edit (pencil) icon, then click Test. This will send a test alert to Grafana IRM

Secure the integration with a bearer token

By default, any request sent to an integration’s URL is accepted. You can additionally require that each request include a Grafana service account bearer token.

Grafana Alerting delivers alerts through a contact point rather than from a client you configure directly, so enabling this for a Grafana Alerting integration takes two steps: add the token to the connected contact point, then turn on the requirement in Grafana IRM.

Requirements

• A Grafana external service account that holds the alert groups: write permission (grafana-irm-app.alert-groups:write).
• A service account token for that account (it starts with glsa_).

Configure the contact point and enable the option

Configure the contact point before enabling the option in Grafana IRM. Otherwise, alerts sent in the window between the two steps are rejected with 403 Forbidden and the notifications are dropped.

1. In Grafana, navigate to Alerting > Notification configuration > Contact points and edit the contact point connected to the integration.
2. Under the integration’s Optional Grafana IRM settings, set the Authentication Header - Scheme to Bearer and the Authentication Header - Credentials to the service account token, then save.
3. In Grafana IRM, open the integration, go to Integration Settings, turn on Require a Grafana service account token to send alerts to this integration, and click Update Integration.

After both steps are complete, Grafana Alerting includes the token on every request and Grafana IRM rejects any request that is missing a valid token. If the token is later revoked or expires, alerts from Grafana Alerting are rejected with 403 Forbidden, so revoke or rotate the token deliberately.

Note about grouping and autoresolution

Grafana IRM relies on the Grafana Alerting grouping and autoresolution mechanism to ensure consistency between alert state in IRM and AlertManager. It’s recommended to configure grouping on the Grafana Alerting side and use default grouping and autoresolution templates on the IRM side. Changing this templates might lead to incorrect grouping and autoresolution behavior.

Note about legacy integration

Before we were using each alert from Grafana Alerting group as a separate payload:

JSON Copy

{
  "labels": {
    "severity": "critical",
    "alertname": "InstanceDown"
  },
  "annotations": {
    "title": "Instance localhost:8081 down",
    "description": "Node has been down for more than 1 minute"
  },
  ...
}

This behavior was leading to mismatch in alert state between IRM and Grafana Alerting and draining of rate-limits, since each Grafana Alerting alert was counted separately.

We decided to change this behavior to respect Grafana Alerting grouping by using AlertManager group as one payload.

JSON Copy

{
  "alerts": [...],
  "groupLabels": {
    "alertname": "InstanceDown"
  },
  "commonLabels": {
    "job": "node",
    "alertname": "InstanceDown"
  },
  "commonAnnotations": {
    "description": "Node has been down for more than 1 minute"
  },
  "groupKey": "{}:{alertname=\"InstanceDown\"}",
  ...
}

You can read more about AlertManager Data model here.

After-migration checklist

1. Send a new demo alert to the migrated integration
2. Adjust routes to the new shape of payload. You can use payload of the demo alert from previous step as an example
3. If outgoing webhooks utilized the alerts payload from the migrated integration, it’s needed to adjust them as well

Related topics

• Grafana IRM integrations overview
• Alert escalation and routing
• Grafana Alerting documentation