Outgoing webhooks for Grafana IRM
Outgoing webhooks in Grafana IRM enable flexible integration with external systems by automatically sending event data related to alerts and incidents to your preferred tools and APIs.
Note
Grafana IRM outgoing webhooks are now managed from a unified interface and processed through a shared backend service. If you have existing outgoing webhooks configured through the legacy Incident outgoing webhooks integration, those webhooks remain fully functional. However, we recommend migrating to the shared IRM webhook experience to take advantage of the latest features and ensure long-term support. For more information, refer to the Migrate from Incident webhooks section of this document.
Configure an outgoing webhook
To create a webhook from the IRM UI:
- In Grafana Cloud, go to Alerts & IRM > IRM > Integrations.
- Select the Outgoing Webhooks tab.
- Click + Create an Outgoing Webhook.
- Choose a preset:
- Simple – Sends basic alert group data (used for escalation steps).
- Advanced webhook for alert groups – Customizable with additional settings and templating.
- Advanced webhook for incidents – Customizable with additional settings and templating.
- Grafana Sift for alert groups – Forwards alert group data to Grafana Sift.
- Complete the webhook configuration form.
- Click Create to save the webhook.
For information about configuring outgoing webhooks via Terraform, refer to the Terraform documentation.
Webhook configuration fields
The following fields are available when configuring an outgoing webhook:
Field | Description | Required |
---|---|---|
Name | A descriptive display name | Yes |
Enabled | Whether the webhook is active | Yes (default: true) |
Assign to team | Team that owns or manages the webhook | No |
Trigger type | Event that causes the webhook to execute | Yes |
HTTP method | GET, POST, PUT, PATCH, DELETE, or OPTIONS | Yes (default: POST) |
Integrations | Restrict execution to specific integrations | No |
Labels | Labels to include in the payload body | No |
Webhook URL | Destination for the request | Yes |
Headers | Optional request headers (templating supported) | No |
Username | HTTP Basic Auth username | No |
Password | HTTP Basic Auth password | No |
Authorization | Supports basic auth or custom headers | No |
Trigger template | Used to conditionally execute the webhook | No |
Forward whole payload | Sends the full event object as received by IRM instead of a custom body | No |
Note
Your destination endpoints must respond within a 4-second timeout window.
Supported triggers
The following event types are available to trigger a webhook to execute.
Alert group events
Event | Description |
---|---|
Manual or escalation step | Triggered manually or as part of an escalation chain |
Alert group created | A new alert group is created |
Acknowledged | The alert group was acknowledged |
Resolved | The alert group was resolved |
Silenced | The alert group was silenced |
Unsilenced | The alert group was unsilenced |
Unresolved | The alert group was reopened |
Unacknowledged | Acknowledgment was removed |
Status change | Any state change event |
Resolution note added | A resolution note was added |
Personal notification | Triggered by a personal notification rule |
Incident events
Event | Description |
---|---|
Incident declared | A new incident is declared |
Incident changed | Updates to title, severity, status, role, or labels |
Incident resolved | The incident is resolved |
Manual from incident | Trigged manually via dropdown in incident Activity->Action Tab |
Note
Each matching event triggers a webhook once. Removing and re-adding a matching label won’t retrigger the webhook.
Incident subevents
Trigger | Subevent |
---|---|
Incident declared | grafana.incident.created |
Incident changed | grafana.incident.updated.role grafana.incident.updated.status grafana.incident.updated.title grafana.incident.updated.severity grafana.incident.added.label grafana.incident.removed.label |
Incident resolved | grafana.incident.closed |
Manual from incident | grafana.incident.manual |
Use webhook templates
Use Jinja2 templates to include dynamic content from the event context. Templates have access to context data from the triggering event. Supported fields vary by trigger type.
For more information about templates in IRM, refer to the Templates documentation.
Common template fields
{{ event.type }}
– Type of event{{ event.time }}
– Timestamp{{ user.username }}
– Username if triggered by a user{{ alert_group.title }}
– Alert group title{{ alert_payload }}
– Contents of the first alert in the group{{ incident.title }}
– Incident title{{ incident.severity }}
– Incident severity{{ webhook.name }}
– Webhook name
Available template fields for alert groups
The following fields are available to templates when a webhook executes:
Event information
{{ event.type }}
- Type of event that triggered the webhook{{ event.time }}
- Timestamp when the event occurred{{ event.user.* }}
- Context data as provided by the user for Personal Notification webhooks
User information
Available when the event was triggered by a user action:
{{ user.id }}
- User ID{{ user.username }}
- Username{{ user.email }}
- Email address
Alert group information
{{ alert_group.id }}
- Alert group ID{{ alert_group.integration_id }}
- Integration ID{{ alert_group.route_id }}
- Route ID{{ alert_group.alerts_count }}
- Number of alerts in the group{{ alert_group.state }}
- Current state{{ alert_group.created_at }}
- Creation timestamp{{ alert_group.resolved_at }}
- Resolution timestamp{{ alert_group.acknowledged_at }}
- Acknowledgment timestamp{{ alert_group.title }}
- Alert group title{{ alert_group.permalinks }}
- Links to the alert group{{ alert_group.labels }}
- Alert group labels{{ alert_group.resolution_notes }}
- Alert group resolution notes
Alert information
{{ alert_payload }}
- Content of the first alert in the group
Integration information
{{ integration.id }}
- Integration ID{{ integration.type }}
- Integration type{{ integration.name }}
- Integration name{{ integration.team }}
- Team the integration belongs to{{ integration.labels }}
- Integration labels
User action information
{{ alert_group_acknowledged_by }}
- User who acknowledged the alert{{ alert_group_resolved_by }}
- User who resolved the alert
Notification information
{{ notified_users }}
- Array of users who received notifications{{ users_to_notify }}
- Array of users who will be notified next
Webhook information
{{ webhook.id }}
- Webhook ID{{ webhook.name }}
- Webhook name{{ webhook.labels }}
- Webhook labels
Previous webhook responses
{{ responses }}
- Responses from other webhooks for this alert group
Available template fields for incidents
The following fields are available to templates when a webhook executes:
Event information
{{ event.type }}
- Type of event that triggered the webhook{{ event.event }}
- Specific incident event (e.g.grafana.incident.updated.title
){{ event.time }}
- Timestamp when the event occurred
Incident information
{{ incident.incidentID }}
- Incident ID{{ incident.title }}
- Title of the incident{{ incident.summary }}
- Summary of the incident{{ incident.status }}
- Status of the incident{{ incident.severity }}
- Severity of the incident{{ incident.labels }}
- Incident labels{{ incident.incidentStart }}
- Incident start timestamp{{ incident.incidentEnd }}
- Incident end timestamp
For a detailed list of all the available fields refer to the Incident API reference documentation.
Webhook information
{{ webhook.id }}
- Webhook ID{{ webhook.name }}
- Webhook name{{ webhook.labels }}
- Webhook labels
Previous webhook responses
{{ responses }}
- Responses from other webhooks for this incident
Example templates
Basic JSON payload:
{
"name": "{{ alert_payload.labels.alertname }}",
"message": "{{ alert_payload.annotations.description }}"
}
URL with query params:
https://example.com/tickets?assignee={{ user.email }}
Use tojson()
to format objects:
{
"labels": {{ alert_payload.labels | tojson() }}
}
Reference previous responses
You can access results from previously executed webhooks using the {{ responses }}
variable.
This enables advanced workflows, such as updating a ticket that was created by an earlier webhook.
Note
There is a retention policy of 90 days for previous responses.
Conditional execution
Use the Trigger Template field to control when a webhook executes. The webhook will run only if the template evaluates to an empty string, True
, or 1
.
Example to trigger only for high-severity alerts:
{% if alert_payload.labels.severity == "high" %}true{% endif %}
Example to trigger only for incidents with “Critical” severity and the “Customer” label set:
{% if incident.severity == "Critical" and incident.labels | selectattr('key', 'equalto', 'Customer') | list | length > 0 %}true{% endif %}
View webhook execution results
You can view webhook execution history in the Outgoing Webhooks tab. The list includes the latest events and associated data.
Click a webhook to view detailed execution information:
- The triggered event and context
- Request details: URL, headers, and body
- Response code and response body
- Template evaluation output
You can also find webhook responses associated with specific events:
- For alert group webhooks, a link to the corresponding response appears in the alert group timeline, accessible from the alert group detail page.
- For incident webhooks, execution results appear in the incident timeline under the All activity view.
Migrate from Incident outgoing webhooks
Legacy incident webhooks continue to work but will eventually be deprecated in future releases.
To migrate a webhook from the legacy Incident outgoing webhooks app, manually recreate the webhook in the shared IRM Outgoing Webhooks interface with the following settings:
Name: <your webhook name>
HTTP method: POST
URL: <your URL>
Trigger type: <your original trigger type>
Customize forwarded data [x]
Data template: {{ incident }}
Be aware that the IRM shared webhook requests do not include signature or Cloud Events headers.
After enabling the corresponding IRM webhook, test your new webhook before removing the legacy webhook. Once verified, delete the legacy configuration to avoid duplicate executions.
If your webhook previously used the Incident matches filters trigger, use the Incident changed trigger type. If you want to respond to other incident events, such as declarations or resolutions, create additional webhooks for those trigger types as needed.
Recreate your filter logic using Jinja templating in the Trigger template field. This provides equivalent behavior to the original filtered trigger in the legacy Incident integration.
For example:
-severity:critical
is equivalent to {{ incident.severity != "critical" }}
or
and(label:customer severity:critical)
can be expressed as {{ incident.severity == "Critical" and incident.labels | selectattr('key', 'equalto', 'Customer') | list | length > 0 }}
Examples and integrations
Outgoing webhooks work well in conjunction with the IRM API.
For example webhook configurations with common integrations, refer to: