Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentationbreadcrumb arrow Grafana OnCallbreadcrumb arrow Connect to Grafana OnCallbreadcrumb arrow Use webhooks to send and receive alertsbreadcrumb arrow Format alerts with templates
Open source

Format alerts with templates

Grafana OnCall works with over one thousand alert monitoring systems. Almost any monitoring system can send alerts using webhooks with JSON payloads.

By default, webhooks will deliver raw JSON. To modify the payload to be more human-readable, you can format your alerts fields that OnCall recognizes. You can use Jinja templates for more advanced customization.

JSON alerting object

Alerts we receive contain metadata as keys and values in a JSON object. The following is an example of an alert from Grafana:

json 
{
  "dashboardId":1,
  "title":"[Alerting] Panel Title alert",
  "message":"Notification Message",
  "evalMatches":[
    {
      "value":1,
      "metric":"Count",
      "tags":{}
    }
  ],
  "imageUrl":"https://grafana.com/static/assets/img/blog/mixed_styles.png",
  "orgId":1,
  "panelId":2,
  "ruleId":1,
  "ruleName":"Panel Title alert",
  "ruleUrl":"http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
  "state":"alerting",
  "tags":{
    "tag name":"tag value"
  }
}

The alert payload

Once an alert is received by Grafana OnCall, the following occurs, based on the alert content:

  1. The most useful information is shown in a readable format.

  2. Noise is minimized by grouping alerts, combining similar alerts into a single page.

  3. The alert group is resolved if the monitoring system tells Grafana OnCall to do so.

In Grafana OnCall every alert and alert group has the following fields:

  • Title, message and image url
  • Grouping Id
  • Resolve Signal

The JSON payload is converted. For example:

  • {{ payload.title }} -> Title
  • {{ payload.message }} -> Message
  • {{ payload.imageUrl }} -> Image Url

The result is that each field of the alert in OnCall is now mapped to the JSON payload keys. This also true for the alert behavior:

  • {{ payload.ruleId }} -> Grouping Id
  • {{ 1 if payload.state == 'OK' else 0 }} -> Resolve Signal

OnCall has default Jinja templates for the most popular monitoring systems.

If your monitoring system is not in the Grafana OnCAll integrations list you can create the most generic integration Webhook, send an alert, and write your own templates.

As a best practice, add _Playbooks_, _Useful links_, or _Checklists_ to the alert message.

How to customize templates

You can customize the default templates in Grafana OnCall by opening the Settings window in either the Integrations or Alert Groups tab:

  1. From the Integrations tab, select the integration, then click the Settings (gear) icon.

  2. From the Alert Groups tab, click Edit rendering, grouping, and other templates

  3. In Settings, select the template to edit from Edit template for.

  4. Edit the Appearances template as needed:

    • Title, Message, Image url for Web
    • Title, Message, Image url for Slack
    • Title used in SMS
    • Title used in Phone
    • Title, Message used in Email

  5. Edit the alert behavior as needed:

    • Grouping Id - This output groups other alerts into a single alert group.
    • Acknowledge Condition - The output should be ok, true, or 1 to auto-acknowledge the alert group. For example, {{ 1 if payload.state == 'OK' else 0 }}.
    • Resolve Condition - The output should be ok, true or 1 to auto-resolve the alert group. For example, {{ 1 if payload.state == 'OK' else 0 }}.
    • Source Link - Used to customize the URL link to provide as the “source” of the alert.

Advanced Jinja templates

Grafana OnCall uses Jinja templating language to format alert groups for the Web, Slack, phone calls, SMS messages, and more because the JSON format is not easily readable by humans. As a result, you can decide what you want to see when an alert group is triggered as well as how it should be presented.

Jinja2 offers simple but multi-faceted functionality by using loops, conditions, functions, and more.

NOTE: Every alert from a monitoring system comes in the key/value format. Grafana OnCall has rules about which of the keys match to: __title, message, image, grouping, and auto-resolve__.

Loops

Monitoring systems can send an array of values. In this example, you can use Jinja to iterate and format the alert using a Grafana example:

.jinja2 
*Values:*
 {% for evalMatch in payload.evalMatches -%}
 `{{ evalMatch['metric'] }}: '{{ evalMatch['value'] -}}'`{{ " " }}
 {%- endfor %}

Conditions

You can add instructions if an alert comes from a specified Grafana alert rule:

jinja2 
{% if  payload.ruleId == '1' -%}
*Alert TODOs*
1. Get acess to the container
    ```
        kubectl port-forward service/example 3000:80
    ```
2. Check for the exception.
3. Open the container and reload caches.
4. Click Custom Button `Send to Jira`
{%- endif -%}

Built-in Jinja functions

Jinja2 includes built-in functions that can also be used in Grafana OnCall. For example:

.jinja2 
{{ payload | tojson_pretty }}

Built-in functions:

  • abs
  • capitalize
  • trim
  • You can see the full list of Jinja built-in functions on github here

Functions added by Grafana OnCall

  • time - current time
  • tojson_pretty - JSON prettified
  • iso8601_to_time - converts time from iso8601 (2015-02-17T18:30:20.000Z) to datetime
  • datetimeformat - converts time from datetime to the given format (%H:%M / %d-%m-%Y by default)

Related documentation