Menu
Grafana Cloud Grafana Incident Grafana Incident APIs Grafana Incident JSON/HTTP RPC API Incoming Webhook
Grafana Cloud

Note: This feature is currently experimental or under active development. Some details may change when the feature is released.

Incoming Webhook

The Incoming Webhook allows you to wire up any third-party system to trigger an Incident.

Details of the webhook are attached to the Incident.

Get started

It takes only a few steps to configure a third-party tool to automatically declare an Incident.

Create a dedicated Service Account

You will need to provide the third-party webhook with a Service account token in order to access the endpoint.

It is recommended that you use a dedicated service account for your Incoming Webhooks. You may want to revoke access as a safety measure should any of the keys get abused. See Disabling an Incoming Webhook below.

For more information about creating Service accounts and Service account tokens, see Authenticating the Grafana Incident APIs.

Once you have your Service account token you should configure the third-party to send the following HTTP header inside the POST request:

Authorization: Bearer glsa_HOruNAb7SOiCdshU9algkrq7F...

Build the URL

Replace your-stack.grafana.net with your own instance in the following URL:

https://your-stack.grafana.net/api/plugins/grafana-incident-app/resources/api/experimental/incoming-webhook/grafana.incident.create

Use URL parameters to customize the Incident

You can add the following parameters to the URL to further customize the Incident.

  • drill=true (boolean) If true will declare a drill (test) Incident (recommended for testing)
  • title - (string) The title of the Incident
  • severity - (string) The severity of the Incident to declare
  • labels - (list of string) A comma-separated list of labels to add to the Incident
  • roomprefix - (string) The chat room prefix for platforms that support it (e.g. incident)

Using special values, you can extract fields from the Webhook’s JSON request bodies.

Paste the URL in the third-party tool

Configure the third party system to issue a POST request to the incoming-webhooks/grafana.incident.create endpoint from the previous step.

POST .../api/experimental/incoming-webhooks/grafana.incident.create
  &drill=true                         // make it a drill
  &title=json(message.shortMessage)   // extract the field from the body
  &severity=minor                     // minor severity assumed
  &labels=autogenerated               // label it as 'autogenerated' so we know
Content-Type: application/json

{
  "message": {
    "shortMessage": "A short description might appear here"
  }
}

Only share the Incoming Webhook URL with the third-party system. If it leaks, anybody would be able to declare incidents in your system by making a POST request. Learn more about Disabling an Incoming Webhook

Lookup JSON values from the body

If your Webhook sports a Content-Type: application/json with an object in the body, you can access the data using the json() function in the URL parameters.

  • Only JSON payloads that are within the 1MB limit will be parsed

For example, a Webhook body might contain some data like this:

Content-Type: "application/json"
{
  "reportID": "abc123",
  "report": {
    "title": "Unable to access public website"
  }
}

By adding the title=json(report.title) parameter to our Incoming Webhook URL, we are able to extract the nested report.title:

.../incoming-webhooks/grafana.incident.create?title=json(report.title)

When the Incident is declared via this route, the Incident’s title will match the report.title value “Unable to access public website”.

If it is not possible to parse the JSON and find the value, a sensible default will be used.

Disabling an Incoming Webhook

To stop an Incoming Webhook from working, you should revoke access to the Service account token you created for this Incoming Webhook.

When should you disable a Service account token?

You should immediately delete an Incoming Webhook URL if:

  • the URL has leaked or been otherwise compromised
  • you are no longer using the Incoming Webhook URL
  • you notice any suspicious activity
  • the third-party system starts declaring too many Incidents