Incoming Webhooks for Grafana Incident

The Incoming Webhook integration allows you to wire up any third-party system to trigger an Incident.

Details of the webhook are attached to the Incident.

Get started

It takes only a few steps to configure a third-party tool to automatically declare an Incident.

Enable Incoming Webhooks

In the Grafana Incident web app, an admin can go to Integrations to enable incoming webhooks.

Installing the Incoming Webhooks integration will generate a token which you will use to authorize the requests.

Go to Alerts & Incidents and under Incident choose Integrations
Select the Incoming Webhooks integration
Click Install integration
Make note of the Token and URL fields

Get the URL

Copy the URL and the token from the integration details.

There are two ways to pass the token in the request:

You can set the Authorization header (recommended) to Bearer {token}
You can add the token as a URL parameter

Use URL parameters to customize the incident

You can add the following parameters to the URL to further customize the Incident.

drill=true (boolean) If true will declare a drill (test) incident (recommended for testing)
title - (string) Required. The title of the incident
severity - (string) The severity of the incident to declare (default: pending)
status - (string) The status of the incident, either active or resolved (default: active)
labels - (list of string) A comma-separated list of labels to add to the incident
roomprefix - (string) The chat room prefix for platforms that support it (e.g. incident)
url - (string) A link to some context, usually the place (or thing) where the incident was declared
caption - (string) An optional caption to go along with url

You can also decide what the response should include:

includes - (comma separated list) A list of fields to include in the response, e.g. ?includes=incident will return the incident object.

Using special values, you can extract fields from the Webhook’s JSON request bodies.

Paste the URL into the third-party tool

Configure the third party system to issue a POST request to the api/experimental/incoming-webhooks/grafana.incident.create endpoint from the previous step.

POST .../api/v1/incoming-webhooks/grafana.incident.create
  &drill=true                         // make it a drill
  &title=json(message.shortMessage)   // extract the field from the body
  &severity=minor                     // minor severity assumed
  &labels=autogenerated               // label it as 'autogenerated' so we know
Content-Type: application/json
Authorization: Bearer token_goes_here

{
  "message": {
    "shortMessage": "A short description might appear here"
  }
}

Only share the Incoming Webhook token with the third-party system. If it leaks, anybody would be able to declare incidents in your system by making a POST request. Learn more about Disabling an Incoming Webhook

Understanding the response

In most cases, the handler will return a 200 OK indicating success. The body of the webhook request will be the IncomingWebhookResponse JSON object:

{
  "incident": {
    "incidentID": "incident-1",
    "title": "title of the incident"
    // more incident fields
  },
  "processingErrors": ["something went wrong"]
}

It is possible for errors to occur during processing, but the handler errs on the side of always declaring the incident, regardless. You can check the processingErrors field in the response for any issues during development. You should resolve all errors before shipping to production.

Test the endpoint using curl

The integration details provides you with an example curl command to test the endpoint.

Or you can write your own, remembering to use your own token:

curl ".../api/v1/incoming-webhook/grafana.incident.create?drill=true&title=json(report.title)" \
  --request POST
  --header 'Authorization: Bearer use_real_token'
  --data '{"report":{"title":"testing the new incoming webhooks integration"}}'

Notice that a drill incident has been declared, with details of the request included in the timeline.

Lookup JSON values from the body

If your Webhook sports a Content-Type: application/json with an object in the body, you can access the data using the json() function in the URL parameters.

Only JSON payloads that are within the 1MB limit will be parsed

For example, a Webhook body might contain some data like this:

Content-Type: "application/json"
{
  "reportID": "abc123",
  "report": {
    "title": "Unable to access public website"
  }
}

By adding the title=json(report.title) parameter to our Incoming Webhook URL, we are able to extract the nested report.title:

.../api/v1/incoming-webhooks/grafana.incident.create?title=json(report.title)

When the incident is declared via this route, the incident’s title will match the report.title value “Unable to access public website”.

If it is not possible to parse the JSON and find the value, a sensible default will be used.

Disabling Incoming Webhooks

To stop an Incoming Webhook from working, you just reinstall the integration and a new token will be generated, rendering the previous one useless.

Go to Alerts & Incidents and under Incident choose Integrations
Select the Incoming Webhooks integration
Click Uninstall - this will stop accepting any incoming webhook requests
Click Install integration - a new token will be generated
Make note of the Token and URL fields

When should you reinstall Incoming Webhooks?

You should reset your tokens by reinstalling your Incoming Webhooks integration if:

the URL has leaked or been otherwise compromised
you are no longer using the Incoming Webhook URL
you notice any suspicious activity
the third-party system starts declaring too many incidents

Add metadata to your incoming webhooks

You can optionally add metadata, which will be included in the incident, via the following HTTP headers:

User-Agent
Origin
Referer
Link

These values are not processed, they are just repeated inside the incident, but can be useful ways to identify the source of the webhook.

Usage limits

Only the first 1MB of the webhook body will be read - so JSON payloads must be smaller than 1MB
You may only declare an incident via an incoming webhook once per minute. 429 Too Many Requests will be returned if you exceed this limit.