Use webhooks to send and receive alerts on Grafana Labs

Webhook integration

Tue, 11 Mar 2025 08:38:14 -0600

Integrate with your data source using webhooks

Grafana OnCall directly supports integrations from many data sources, but you can connect to any data source that isn’t listed in the Create Integration page by using webhooks.

In Integrations, click + New integration for receiving alerts.
Select a webhook format. There are two available formats. Webhook and Formatted Webhook.
- Webhook will pull all of the raw JSON information and display it in the manner that it is received.
- Formatted Webhook can be used if the body of the alerts sent by your monitoring service are formatted in a way that OnCall can read. The following fields are recognized, but none are required:
  - alert_uid: a unique alert ID for grouping.
  - title: a title.
  - image_url: a URL for an image attached to alert.
  - state: either ok or alerting. Helpful for auto-resolving.
  - link_to_upstream_details: link back to your monitoring system.
  - message: alert details.

Use the unique webhook URL for requests. For example:

curl -X POST \
https://a-prod-us-central-0.grafana.net/integrations/v1/formatted_webhook/m12xmIjOcgwH74UF8CN4dk0Dh/ \
-H 'Content-Type: Application/json' \
-d '{
    "alert_uid": "08d6891a-835c-e661-39fa-96b6a9e26552",
    "title": "The whole system is down",
    "image_url": "https://upload.wikimedia.org/wikipedia/commons/e/ee/Grumpy_Cat_by_Gage_Skidmore.jpg",
    "state": "alerting",
    "link_to_upstream_details": "https://en.wikipedia.org/wiki/Downtime",
    "message": "Smth happened. Oh no!"
}'
    ```

Format alerts with templates

Tue, 11 Mar 2025 08:38:14 -0600

Format alerts with templates

Grafana OnCall works with over one thousand alert monitoring systems. Almost any monitoring system can send alerts using webhooks with JSON payloads.

By default, webhooks will deliver raw JSON. To modify the payload to be more human-readable, you can format your alerts fields that OnCall recognizes. You can use Jinja templates for more advanced customization.

JSON alerting object

Alerts we receive contain metadata as keys and values in a JSON object. The following is an example of an alert from Grafana:

{
  "dashboardId":1,
  "title":"[Alerting] Panel Title alert",
  "message":"Notification Message",
  "evalMatches":[
    {
      "value":1,
      "metric":"Count",
      "tags":{}
    }
  ],
  "imageUrl":"https://grafana.com/static/assets/img/blog/mixed_styles.png",
  "orgId":1,
  "panelId":2,
  "ruleId":1,
  "ruleName":"Panel Title alert",
  "ruleUrl":"http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
  "state":"alerting",
  "tags":{
    "tag name":"tag value"
  }
}

The alert payload

Once an alert is received by Grafana OnCall, the following occurs, based on the alert content:

The most useful information is shown in a readable format.
Noise is minimized by grouping alerts, combining similar alerts into a single page.
The alert group is resolved if the monitoring system tells Grafana OnCall to do so.

In Grafana OnCall every alert and alert group has the following fields:

Title, message and image url
Grouping Id
Resolve Signal

The JSON payload is converted. For example:

{{ payload.title }} -> Title
{{ payload.message }} -> Message
{{ payload.imageUrl }} -> Image Url

The result is that each field of the alert in OnCall is now mapped to the JSON payload keys. This also true for the alert behavior:

{{ payload.ruleId }} -> Grouping Id
{{ 1 if payload.state == 'OK' else 0 }} -> Resolve Signal

OnCall has default Jinja templates for the most popular monitoring systems.

If your monitoring system is not in the Grafana OnCAll integrations list you can create the most generic integration Webhook, send an alert, and write your own templates.

As a best practice, add _Playbooks_, _Useful links_, or _Checklists_ to the alert message.

How to customize templates

You can customize the default templates in Grafana OnCall by opening the Settings window in either the Integrations or Alert Groups tab:

From the Integrations tab, select the integration, then click the Settings (gear) icon.
From the Alert Groups tab, click Edit rendering, grouping, and other templates
In Settings, select the template to edit from Edit template for.
Edit the Appearances template as needed:
- Title, Message, Image url for Web
- Title, Message, Image url for Slack
- Title used in SMS
- Title used in Phone
- Title, Message used in Email
Edit the alert behavior as needed:
- Grouping Id - This output groups other alerts into a single alert group.
- Acknowledge Condition - The output should be ok, true, or 1 to auto-acknowledge the alert group. For example, {{ 1 if payload.state == 'OK' else 0 }}.
- Resolve Condition - The output should be ok, true or 1 to auto-resolve the alert group. For example, {{ 1 if payload.state == 'OK' else 0 }}.
- Source Link - Used to customize the URL link to provide as the “source” of the alert.

Advanced Jinja templates

Grafana OnCall uses Jinja templating language to format alert groups for the Web, Slack, phone calls, SMS messages, and more because the JSON format is not easily readable by humans. As a result, you can decide what you want to see when an alert group is triggered as well as how it should be presented.

Jinja2 offers simple but multi-faceted functionality by using loops, conditions, functions, and more.

NOTE: Every alert from a monitoring system comes in the key/value format. Grafana OnCall has rules about which of the keys match to: __title, message, image, grouping, and auto-resolve__.

Loops

Monitoring systems can send an array of values. In this example, you can use Jinja to iterate and format the alert using a Grafana example:

*Values:*
 {% for evalMatch in payload.evalMatches -%}
 `{{ evalMatch['metric'] }}: '{{ evalMatch['value'] -}}'`{{ " " }}
 {%- endfor %}

Conditions

You can add instructions if an alert comes from a specified Grafana alert rule:

{% if  payload.ruleId == '1' -%}
*Alert TODOs*
1. Get acess to the container
    ```
        kubectl port-forward service/example 3000:80
    ```
2. Check for the exception.
3. Open the container and reload caches.
4. Click Custom Button `Send to Jira`
{%- endif -%}

Built-in Jinja functions

Jinja2 includes built-in functions that can also be used in Grafana OnCall. For example:

{{ payload | tojson_pretty }}

Built-in functions:

abs
capitalize
trim
You can see the full list of Jinja built-in functions on github here

Functions added by Grafana OnCall

time - current time
tojson_pretty - JSON prettified
iso8601_to_time - converts time from iso8601 (2015-02-17T18:30:20.000Z) to datetime
datetimeformat - converts time from datetime to the given format (%H:%M / %d-%m-%Y by default)

Send alert notifications by webhook

Tue, 11 Mar 2025 08:38:14 -0600

Send alert group notifications by webhook

You can configure outgoing webhooks to send alerts to destination. Once a webhook is created, you can choose the webhook as a notification method in escalation steps.

In Grafana OnCall, navigate to Outgoing Webhooks and click + Create. This is also the place to edit and delete existing webhooks.
Name your webhook and enter the destination URL.
If the destination requires authentication, enter your credentials. You can enter a username and password (HTTP) or an authorization header formatted in JSON.
Configure the webhook payload in the Data field. You can use four variables to automate the body of your webhook. The format you use to call the variables must match the structure of how the fields are nested in your alert payload. The Data field can use the following four variables to auto-populate the webhook payload with information about the first alert in the alert group:
- {{ alert_title }}
- {{ alert_message }}
- {{ alert_url }}
- {{ alert_payload }}
alert_payload is always the first level of any variable you want to call.

The following is an example of an entry in the Data field that might return an alert name and description.
JSON
```
{
"name": "{{ alert_payload.labels.alertname }}",
"message": "{{ alert_payload.annotations.description }}"
}
```
NOTE: If you get an error message and cannot create a webhook, make sure your JSON is formatted correctly.
Click Create Webhook.