Documentation Grafana Cloud AI and machine learning Grafana Assistant Introduction Set up IRM webhooks

Set up IRM webhooks for automated investigations

Grafana Assistant Investigations can automatically start when alert groups are created or incidents are declared in Grafana IRM. This guide shows you how to configure webhooks so investigations begin working on problems immediately, providing context and insights before human responders arrive.

What you’ll achieve

• Configure webhooks to automatically trigger investigations from IRM alert groups and incidents.
• Understand how webhooks connect IRM events to Assistant investigations.
• Manage webhook configuration and troubleshoot common issues.

Before you begin

• IRM OnCall access: You need access to Grafana IRM OnCall to configure webhooks.
• Required permissions: You need the following RBAC permissions to configure webhooks:
  • grafana-irm-app.outgoing-webhooks:read and grafana-irm-app.outgoing-webhooks:write
  • grafana-irm-app.integrations:read
  • grafana-irm-app.alert-groups:read and grafana-irm-app.alert-groups:write
  • plugins.app:access with scope plugins:id:grafana-irm-app
• IRM integrations: Ensure you have IRM integrations configured for the alert sources you want to monitor. For more information, refer to Integrations in Grafana IRM.

What IRM webhooks do

IRM webhooks create a bridge between Grafana IRM and Grafana Assistant Investigations. When configured, they automatically:

• Trigger investigations when alert groups are created or their status changes
• Trigger investigations when incidents are declared or updated
• Post investigation links back to alert groups and incidents
• Update context when alerts or incidents change, keeping investigations synchronized

This automation ensures investigations start immediately when problems occur, giving you a head start on analysis while you coordinate the incident response.

How webhooks work

When an alert group is created or an incident is declared, IRM sends a webhook notification to Grafana Assistant. The Assistant then:

1. Creates a new investigation with context from the alert or incident
2. Posts a resolution note or key update back to the alert group or incident page
3. Updates the investigation context if the alert or incident changes
4. Prevents duplicate investigations if the same alert triggers multiple times

The investigation workspace includes all relevant context from the alert or incident, including labels, annotations, and status information.

Set up webhooks

You can configure IRM webhooks in three ways, depending on your preferences and access level:

Method 1: Assistant Settings page (recommended)

The Assistant provides a dedicated settings page that simplifies webhook creation and management.

Set up incident webhooks

1. Navigate to Assistant → Settings (from the navigation bar) → IRM webhook setup.
2. In the Add incident webhook section, click Enable incident webhooks.
   • If webhooks are already enabled, the button shows Incident webhooks enabled and is disabled.
   • This automatically creates a webhook that triggers when incidents are updated.

Set up alert group webhooks

1. Navigate to Assistant → Settings → IRM webhook setup.
2. In the Add alert group webhook section:
   • Select one or more IRM integrations from the dropdown.
   • Enter a descriptive webhook name, such as “Production Alerts” or “Critical Issues”.
   • Click Enable alert group webhooks.
3. The system creates webhooks that trigger when alert groups are created and when their status changes.

The Existing webhooks section lists all Assistant-related IRM webhooks, showing their status, trigger types, and configuration.

Method 2: IRM preset webhook

IRM OnCall includes a built-in Grafana Assistant webhook preset that simplifies setup:

1. Navigate to Alerts & IRM → IRM → Integrations → Outgoing webhooks → New webhook.
2. Select the Grafana Assistant for IRM preset from the dropdown.
3. Choose which integrations should trigger investigations.
4. Select trigger types:
   • For alert group webhooks, choose Alert group created and Status change.
   • For incident webhooks, choose Incident changed.
5. Create one webhook per trigger type.

The preset automatically configures the correct endpoint URL and payload format. For a complete list of available trigger types, refer to Supported triggers. For detailed information about IRM webhook configuration, refer to the IRM outgoing webhooks documentation.

Method 3: Manual webhook configuration

If you need more control over webhook configuration, you can create webhooks manually:

1. Navigate to IRM → Integrations → Outgoing Webhooks → Create an Outgoing Webhook.
2. Configure the webhook:
   • Name: Choose a descriptive name
   • HTTP method: POST
   • URL: The Assistant webhook endpoint URL
   • Trigger type: Select the events that should trigger investigations
     • For alert group webhooks, use Alert group created and Status change
     • For incident webhooks, use Incident changed
   • Integrations: Select which integrations should trigger this webhook

For alert group webhooks, use this endpoint URL format:

Copy

<stack-url>/api/plugins/grafana-assistant-app/resources/api/v1/investigations/from-irm

For incident webhooks, use this endpoint URL format:

Copy

<stack-url>/api/plugins/grafana-assistant-app/resources/api/v1/investigations/from-irm-incident

What happens when webhooks trigger

Alert group webhooks

When an alert group triggers a webhook:

1. Investigation creation: A new investigation is created with the alert group title and context.
2. Resolution note: A resolution note is posted to the alert group UI with a link to the investigation.
3. Slack integration: If Slack is configured, the resolution note also appears as a thread in the Slack message.
4. Context updates: If the alert group status or labels change, the investigation context is updated automatically.

Incident webhooks

When an incident is updated and triggers a webhook:

1. Context check: The system checks if the incident has meaningful context. Investigations only start when sufficient context is available.
2. Investigation creation: If enough context exists, a new investigation is created with the incident title, summary, and activity context.
3. Activity item: An activity item is posted to the incident page informing users that an investigation has started.
4. Completion update: When the investigation finishes, a second activity item is posted with the investigation results.
5. Slack integration: If Slack is configured, the same updates are sent to the incident Slack channel.
6. Context updates: If the incident status, severity, labels, or activity items change, the investigation context is updated automatically with new activity information.

View investigation results

After a webhook triggers an investigation, you can view the results in several places:

• Alert group timeline: Click the investigation link in the resolution note posted to the alert group.
• Incident page: View key updates at the top of the incident page that include investigation links.
• Assistant interface: Open the investigation workspace directly from the Assistant sidebar.
• Slack: Click investigation links in Slack threads (if Slack integration is configured).

Manage webhooks

View existing webhooks

In the Assistant settings page, the Existing webhooks section shows:

• Webhook name and status (enabled or disabled)
• HTTP method and trigger type
• Target URL
• Links to edit webhooks in IRM OnCall

Edit webhooks

To modify webhook configuration:

1. Click the webhook name in the Existing webhooks section, or
2. Navigate directly to IRM OnCall → Integrations → Webhooks and select the webhook to edit.

Disable webhooks

To temporarily disable a webhook without deleting it:

1. Navigate to IRM OnCall → Integrations → Webhooks.
2. Select the webhook you want to disable.
3. Toggle the Enabled switch to off.

Delete webhooks

To permanently remove a webhook:

1. Navigate to IRM OnCall → Integrations → Webhooks.
2. Select the webhook you want to delete.
3. Click Delete and confirm the action.

Best practices

Only enable webhooks for integrations that generate alerts you want to investigate automatically. This prevents unnecessary investigations for low-priority alerts.

Use clear, descriptive names for webhooks so you can easily identify their purpose later.

Start with a few integrations and monitor how many investigations are created. Adjust your configuration based on your team’s capacity.

Automated investigations work best for common, well-understood alert patterns. For complex or novel issues, consider launching manual investigations with more specific context.

Periodically review investigation results to ensure they’re providing value. Adjust webhook triggers or investigation prompts as needed.

Troubleshooting

Webhooks not triggering investigations

Check the webhook status in IRM OnCall to verify it’s enabled.

Ensure the correct integrations are selected for the webhook.

Confirm the webhook is configured for the events you expect (alert group created, incident declared, etc.).

In IRM OnCall, check the webhook execution history to see if requests are being sent and what responses are received.

Missing investigation links

If investigation links don’t appear in alert groups or incidents, verify the webhook URL is correct and points to your Grafana stack. Ensure the webhook has the necessary permissions to post to alert groups and incidents. Check IRM OnCall webhook execution history for errors.

Investigation context not updating

If investigation context doesn’t update when alerts or incidents change, ensure the webhook includes “Status change” or “Incident changed” trigger types.

Review webhook execution history to confirm update events are being sent.

Next steps

• Learn how to Run a systematic investigation to understand the investigation workflow.
• Review Understand Assistant investigations to learn more about how investigations work.
• Explore Configure Assistant rules to customize investigation behavior and responses.