Outgoing Webhook for Grafana Incident

The Outgoing Webhook integration allows you to receive real-time notifications and updates where you need them. It can trigger events when an incident is created, updated, or closed, and Grafana Incident will notify the specified URL.

Use the Outgoing Webhook integration to create custom workflows and actions based on the details of an Incident.

Note: Outgoing Webhooks are useful in conjunction with the other Grafana Incident APIs.

Example use case

Consider the following use case of the Outgoing Webhook integration:

Multiple teams at your organization use Grafana Incident, so you use labels to identify which incidents belong to which teams. Your current incident response workflow includes a manual step to create an issue in the related teams’ Jira project when an incident is declared.

To automate the issue creation in Jira, use the Outgoing Webhook integration to trigger a custom workflow that reads the Incident event labels and uses them to create an issue in the related Jira project for the right team.

Install the Outgoing Webhook integration

An Admin can install the integration:

From the Grafana Incident Integrations tab, select Outgoing Webhook.
Click Install Integration.
Once you install the integration, a signing secret is automatically generated. You will use this secret when verifying the signature in your webhook handler code.

Configure actions

An admin can wire-up event actions to indicate when the Webhook should be fired.

Click Run when an event fires.
Enter a target endpoint URL for the outgoing webhook POST request.
Click Add event action to save your webhook - see options below:

Trigger webhooks based on events

Once configured, you can trigger outgoing webhooks based on different actions. The following actions are supported:

Incident declared: The webhook triggers when an incident is declared.
Incident change: Triggered when there is a change in the incident status or details.
Incident closed: Triggers when an incident is closed
Incident matches a filter: This trigger activates when an incident matches specific filter criteria.

Define filters for event triggers

When setting up filters for event triggers, you can utilize various conditions to specify the criteria. Each condition consists of a key-value pair separated by a colon. Additionally, you can employ logical operators such as and(), or(), and - (negation) to create filter combinations.

Supported filter conditions:

status: active | resolved
isdrill: true | false
severity: critical | major | minor
role: investigator | commander
label: "<label-name>" Quotes are optional
and(): Used to combine multiple conditions with the logical AND operator (default behavior if no operator is specified).
or(): Combines conditions with the logical OR operator.
-: Negates a condition.

Examples:

-severity:critical: Matches incidents with severity other than critical.
and(label:customer severity:critical): Matches critical incidents with the customer label.
(label:customer severity:critical): Simplified form of the above, as and is the default operator.
and(severity:critical or(label:"Customer A" label:"Customer B")): Matches critical incidents with either Customer A or Customer B label.

Event actions will trigger at most once per incident. For instance, if a label:customer filter is applied, adding, removing, and re-adding a customer label will result in only one webhook call.

Available event types

	Event type
Incident is declared	`grafana.incident.created`
Incident changes	`grafana.incident.updated.role` `grafana.incident.updated.status` `grafana.incident.updated.title` `grafana.incident.updated.severity` `grafana.incident.added.label` `grafana.incident.removed.label`
Incident is resolved	`grafana.incident.closed`

Verify webhook requests

The GI-Signature header is included with requests and has a Unix-seconds timestamp (t=) followed by one or more versioned signatures (e.g. v1=) for example:

GI-Signature:t=1677589543,v1=12d6e3e06e66f2a32b4027827fbb95ce139ee8381a1cec1c40a02ec5f877797c

The v1 signature is calculated from the HMAC-SHA256 value of a “signing string” follows:

signing-string = body-hash + ":" +timestamp + ":" + signature-version

For example, given a payload of { 'somejson': true }, a secret of some-secret-value, a target endpoint URL of https://myendpoint.com/some/path sent at 1677589543:

signature-version = "v1"
body-hash = "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM="
signing-string = "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=:1677589543:v1"
hmac = a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3

GI-Signature:t=1677589543,v1=a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3

Verifying on the command line using openssl:

echo -n "{ 'somejson': true }" | openssl dgst -sha256 -binary | openssl enc -base64
Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=

echo -n "Zlr4UlirxgcHRLkhoMEI43NjQxMBWQSVYQ8ZGAABseM=:1677589543:v1" | openssl dgst -sha256 -hmac "some-secret-value"
a012b87c2e32680e8d028fd19b32f23f64f26f1632b6ff91dc17acd4214f27b3

Sample Outgoing Webhook

Below is a sample for an incident with an updated severity (grafana.incident.updated.severity).

Request headers

As described above, the signature found in GI-Signature can be used to verify the event integrity.

Header	Value
gi-signature	t=1677589543,v1=12d6e3e06e66f2a32b4027827fbb95ce139ee8381a1cec1c40a02ec5f877797c
content-type	application/json

To conform with the CloudEvents specification we also provide the following headers (prefixed with ce-):

Header	Value
ce-type	incident.webhook
ce-time	2023-03-02T14:12:00Z
ce-subject	grafana.incident.updated.severity
ce-specversion	1.0
ce-source	/grafana/incident
ce-id	webhook-out-6400aeb030670e95
ce-dataschema	v1.0.0

Webhook payload

The body of the webhook request will be the OutgoingWebhookPayload JSON object:

{
  "version": "v1.0.0",
  "id": "webhook-out-6400aeb030670e95",
  "source": "/grafana/incident",
  "time": "2023-03-02T14:12:00Z",
  "event": "grafana.incident.updated.severity",
  "incident": {
    "incidentID": "5",
    "severity": "critical",
    "labels": [],
    "isDrill": false,
    "createdTime": "2023-02-27T15:06:25.243788Z",
    "modifiedTime": "2023-03-02T14:12:00.730047Z",
    "createdByUser": {
      "userID": "grafana-incident:user-63f8b6204887f793",
      "name": "admin",
      "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
    },
    "closedTime": "",
    "durationSeconds": 860734,
    "status": "active",
    "title": "55",
    "overviewURL": "/a/grafana-incident-app/incidents/5/55",
    "roles": [
      {
        "role": "investigator",
        "description": "Leads the investigation (has their full-time attention)",
        "maxPeople": 1,
        "mandatory": true,
        "important": true,
        "user": {
          "userID": "grafana-incident:user-63f8b6204887f793",
          "name": "admin",
          "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
        }
      },
      {
        "role": "observer",
        "description": "Watching the incident",
        "maxPeople": 0,
        "mandatory": false,
        "important": false,
        "user": {
          "userID": "grafana-incident:user-63f8b6204887f793",
          "name": "admin",
          "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
        }
      },
      {
        "role": "commander",
        "description": "Owns the incident (has their full-time attention)",
        "maxPeople": 1,
        "mandatory": true,
        "important": true,
        "user": {
          "userID": "grafana-incident:user-63f8b6204887f793",
          "name": "admin",
          "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
        }
      }
    ],
    "taskList": {
      "tasks": [
        {
          "taskID": "assign-role-investigator",
          "immutable": true,
          "createdTime": "2023-02-27T15:06:25.232016511Z",
          "modifiedTime": "2023-03-01T09:33:24.566202085Z",
          "text": "Assign INVESTIGATOR role",
          "status": "done",
          "authorUser": null,
          "assignedUser": null
        },
        {
          "taskID": "assign-role-commander",
          "immutable": true,
          "createdTime": "2023-02-27T15:06:25.23202747Z",
          "modifiedTime": "2023-03-01T09:33:34.67823384Z",
          "text": "Assign COMMANDER role",
          "status": "done",
          "authorUser": null,
          "assignedUser": null
        },
        {
          "taskID": "must-choose-severity",
          "immutable": true,
          "createdTime": "2023-02-27T15:06:25.232032261Z",
          "modifiedTime": "2023-03-02T14:12:00.729522804Z",
          "text": "Specify incident severity",
          "status": "done",
          "authorUser": null,
          "assignedUser": null
        },
        {
          "taskID": "task-63ff1bf465deb38c",
          "immutable": false,
          "createdTime": "2023-03-01T09:33:40.759215968Z",
          "modifiedTime": "2023-03-01T09:33:55.286168752Z",
          "text": "111",
          "status": "done",
          "authorUser": {
            "userID": "grafana-incident:user-63f8b6204887f793",
            "name": "admin",
            "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
          },
          "assignedUser": {
            "userID": "grafana-incident:user-63f8b6204887f793",
            "name": "admin",
            "photoURL": "https://www.gravatar.com/avatar/46d229b033af06a191ff2267bca9ae56?s=512&d=retro"
          }
        }
      ],
      "todoCount": 0,
      "doneCount": 4
    },
    "summary": "Resolved with no comment",
    "heroImagePath": "/api/hero-images/default_org/mC0B9YIgrBH3Hf9jvnKsvalHT90QtlODeiXDd2aLgRsflEZrRHGS5ld2KfIAbndglMIHiKSR04FPRPBpS34DA182rpeBxF9MSypELoU7nyOzXS09YenaCUtBZEzCi1xd/v1128/5.png",
    "incidentStart": "2023-02-20T15:06:26Z",
    "incidentEnd": ""
  }
}