Documentation Grafana Cloud Platform Assistant Configure IRM webhooks

Configure IRM webhooks for automated investigations

Grafana Assistant Investigations can automatically start when alert groups are created or incidents are declared in Grafana IRM. This guide shows you how to configure webhooks so investigations begin working on problems immediately, providing context and insights before human responders arrive.

What you’ll achieve

• Configure webhooks to automatically trigger investigations from IRM alert groups and incidents.
• Understand how webhooks connect IRM events to Assistant investigations.
• Manage webhook configuration and troubleshoot common issues.

Before you begin

• IRM OnCall access: You need access to Grafana IRM OnCall to configure webhooks.
• Required permissions: You need the following RBAC permissions to configure webhooks:
  • grafana-irm-app.outgoing-webhooks:read and grafana-irm-app.outgoing-webhooks:write
  • grafana-irm-app.integrations:read
  • grafana-irm-app.alert-groups:read and grafana-irm-app.alert-groups:write
  • plugins.app:access with scope plugins:id:grafana-irm-app
• IRM integrations: Ensure you have IRM integrations configured for the alert sources you want to monitor. For more information, refer to Integrations in Grafana IRM.

If you have webhook read access but not grafana-irm-app.outgoing-webhooks:write, you can inspect existing webhooks and alert source coverage, but you can’t create, edit, enable, or disable webhooks. Ask your Grafana administrator for write access to manage them. Investigation settings, such as skipped incident labels, use separate Assistant permissions.

What IRM webhooks do

IRM webhooks create a bridge between Grafana IRM and Grafana Assistant Investigations. When configured, they automatically:

• Trigger investigations when alert groups are created or their status changes
• Trigger investigations when incidents are declared or updated
• Post investigation links back to alert groups and incidents
• Update context when alerts or incidents change, keeping investigations synchronized

This automation ensures investigations start immediately when problems occur, giving you a head start on analysis while you coordinate the incident response.

How webhooks work

When an alert group is created or an incident is declared, IRM sends a webhook notification to Grafana Assistant. The Assistant then:

1. Creates a new investigation with context from the alert or incident
2. Posts a resolution note or key update back to the alert group or incident page
3. Updates the investigation context if the alert or incident changes
4. Prevents duplicate investigations if the same alert triggers multiple times

The investigation workspace includes all relevant context from the alert or incident, including labels, annotations, and status information.

Check alert source coverage

Open Assistant > Settings > Investigations and search for an IRM integration in Is an alert source covered?. Results distinguish Not covered, Configured but disabled, and Covered sources.

For an uncovered source, click Add webhook to open the creation form with that integration selected. Review Integrations, enter a Webhook name, and complete the setup. For a configured source, click Edit to find its existing webhook. A disabled webhook doesn’t provide active coverage, so enable it rather than creating a duplicate.

Set up webhooks

You can configure IRM webhooks in three ways, depending on your preferences and access level:

Use Assistant Settings page (recommended)

The Assistant provides a dedicated settings page that simplifies webhook creation and management.

Set up incident webhooks

1. Navigate to Assistant → Settings (from the navigation bar) → Investigations.
2. In the Add incident webhook section, click Enable incident webhooks.
   • If webhooks are already enabled, the button shows Incident webhooks enabled and is disabled.
   • This automatically creates a webhook that triggers when your team declares an incident.

Configure skipped incident labels (optional)

You can optionally prevent investigations from triggering when an incident has specific labels. To view these rules, you need grafana-assistant-app.settings.investigations:read. To edit them, you also need grafana-assistant-app.settings.investigations:write. The Assistant Admin role includes both permissions. If you use a custom role, add the permissions you need. Without settings read access, you can still work with webhooks if you have the required IRM permissions, but incident exclusions aren’t available.

1. Navigate to Assistant → Settings → Investigations.
2. In Skipped incident labels, turn Enabled on.
3. Add one or more key:value rules (for example, team:platform).
4. Save your changes.

When enabled, incidents that match any configured rule do not trigger new investigations. If an investigation already exists for a matching incident, Assistant cancels it.

Set up alert group webhooks

1. Navigate to Assistant → Settings → Investigations.
2. Click Add webhook to open the Add alert group webhook form:
   • Select one or more IRM integrations from the dropdown.
   • Enter a descriptive webhook name, such as “Production Alerts” or “Critical Issues”.
   • Click Enable alert group webhooks.
3. The system creates webhooks that trigger when alert groups are created and when their status changes.

The Existing webhooks section lists all Assistant-related IRM webhooks, showing their status, trigger types, and configuration.

To grant additional teams access to incident-triggered investigations, choose Specific teams under Additional team access. Search by team name, select the teams, and click Save. The picker groups Your teams first and includes other teams you can read in All teams. You can select a team without being a member. If the results are incomplete, type more of the team’s name to narrow the search. Alert group webhook team pickers support the same search.

Use IRM preset webhook

IRM OnCall includes a built-in Grafana Assistant webhook preset that simplifies setup:

1. Navigate to Alerts & IRM → IRM → Integrations → Outgoing webhooks → New webhook.
2. Select the Grafana Assistant for IRM preset from the dropdown.
3. Choose which integrations should trigger investigations.
4. Select trigger types:
   • For alert group webhooks, choose Alert group created and Status change.
   • For incident webhooks, choose Incident declared.
5. Create one webhook per trigger type.

The preset automatically configures the correct endpoint URL and payload format. For a complete list of available trigger types, refer to Supported triggers. For detailed information about IRM webhook configuration, refer to the IRM outgoing webhooks documentation.

Configure webhooks manually

If you need more control over webhook configuration, you can create webhooks manually:

1. Navigate to IRM → Integrations → Outgoing Webhooks → Create an Outgoing Webhook.
2. Configure the webhook:
   • Name: Choose a descriptive name
   • HTTP method: POST
   • URL: The Assistant webhook endpoint URL
   • Trigger type: Select the events that should trigger investigations
     • For alert group webhooks, use Alert group created and Status change
     • For incident webhooks, use Incident declared
   • Integrations: Select which integrations should trigger this webhook

For alert group webhooks, use this endpoint URL format:

Copy

<STACK_URL>/api/plugins/grafana-assistant-app/resources/api/v1/investigations/from-irm

For incident webhooks, use this endpoint URL format:

Copy

<STACK_URL>/api/plugins/grafana-assistant-app/resources/api/v1/investigations/from-irm-incident

Replace STACK_URL with your Grafana Cloud stack URL.

Automatic webhook deliveries require a service credential, not a signed-in user’s session. If you use a Grafana service account, grant it grafana-assistant-app.investigations:create. This requirement applies to both starting investigations and updating existing ones. Your permission to manage webhooks doesn’t grant the delivery account permission to start investigations. For role details, refer to Manage Assistant access with RBAC.

What happens when webhooks trigger

Alert group webhooks

When an alert group triggers a webhook:

1. Investigation creation: A new investigation is created with the alert group title and context.
2. Resolution note: A resolution note is posted to the alert group UI with a link to the investigation.
3. Slack integration: If Slack is configured, the resolution note also appears as a thread in the Slack message.
4. Context updates: If the alert group status or labels change, the investigation context is updated automatically.

Incident webhooks

When your team declares an incident and triggers a webhook:

1. Investigation creation: A new investigation is created with the incident title, summary, and any available activity context.
2. Activity item: An activity item is posted to the incident page informing users that an investigation has started.
3. Completion update: When the investigation finishes, a second activity item is posted with the investigation results.
4. Slack integration: If Slack is configured, the same updates are sent to the incident Slack channel.
5. Context updates: If the incident status, severity, labels, or activity items change, the investigation context is updated automatically with new activity information.

View investigation results

After a webhook triggers an investigation, you can view the results in several places:

• Alert group timeline: Click the investigation link in the resolution note posted to the alert group.
• Incident page: View key updates at the top of the incident page that include investigation links.
• Assistant interface: Open the investigation workspace directly from the Assistant sidebar.
• Slack: Click investigation links in Slack threads (if Slack integration is configured).

Manage webhooks

View existing webhooks

In the Assistant settings page, the Existing webhooks section shows:

• Webhook name and status (enabled or disabled)
• HTTP method and trigger type
• Target URL
• Links to edit webhooks in IRM OnCall

Edit webhooks

To modify webhook configuration:

1. Click the webhook name in the Existing webhooks section, or
2. Navigate directly to IRM OnCall → Integrations → Webhooks and select the webhook to edit.

Disable webhooks

To temporarily disable a webhook without deleting it:

1. Navigate to IRM OnCall → Integrations → Webhooks.
2. Select the webhook you want to disable.
3. Toggle the Enabled switch to off.

Delete webhooks

To permanently remove a webhook:

1. Navigate to IRM OnCall → Integrations → Webhooks.
2. Select the webhook you want to delete.
3. Click Delete and confirm the action.

Follow best practices

Only enable webhooks for integrations that generate alerts you want to investigate automatically. This prevents unnecessary investigations for low-priority alerts.

Use clear, descriptive names for webhooks so you can easily identify their purpose later.

Start with a few integrations and monitor how many investigations are created. Adjust your configuration based on your team’s capacity.

Automated investigations work best for common, well-understood alert patterns. For complex or novel issues, consider launching manual investigations with more specific context.

Periodically review investigation results to ensure they’re providing value. Adjust webhook triggers or investigation prompts as needed.

Troubleshoot common issues

Webhooks not triggering investigations

Check the webhook status in IRM OnCall to verify it’s enabled.

Ensure the correct integrations are selected for the webhook.

Confirm you configured the webhook for the events you expect, such as Alert group created, Status change, or Incident declared.

In IRM OnCall, check the webhook execution history to see if requests are being sent and what responses are received.

If a delivery returns 403 Forbidden, check its authentication. Use a valid service credential for the same Grafana Cloud stack. If you use a service account, ask your Grafana administrator to grant it grafana-assistant-app.investigations:create. Retry the delivery after correcting the credential or permissions.

Missing investigation links

If investigation links don’t appear in alert groups or incidents, verify the webhook URL is correct and points to your Grafana stack. Ensure the webhook has the necessary permissions to post to alert groups and incidents. Check IRM OnCall webhook execution history for errors.

Investigation context not updating

If investigation context doesn’t update when alerts or incidents change, ensure the webhook includes “Status change” or “Incident changed” trigger types.

Review webhook execution history to confirm update events are being sent.

Next steps

• Assistant Investigations to understand how to work with the investigations you’ve automated.
• Configure Assistant rules to customize investigation behavior and responses.