Grafana OnCallConnect to Grafana OnCallConfigure alert notifications with a custom integration

Creating custom connections with integrations

Grafana OnCall works with 1000+ monitoring systems. Almost any monitoring system can send alerts using webhooks with JSON payloads.

You can also use Jinja templates for advanced customization.

JSON alerting object

Alerts we receive contain metadata as keys and values in a JSON object. Here is an example of an alert from Grafana:

{
  "dashboardId":1,
  "title":"[Alerting] Panel Title alert",
  "message":"Notification Message",
  "evalMatches":[
    {
      "value":1,
      "metric":"Count",
      "tags":{}
    }
  ],
  "imageUrl":"https://grafana.com/static/assets/img/blog/mixed_styles.png",
  "orgId":1,
  "panelId":2,
  "ruleId":1,
  "ruleName":"Panel Title alert",
  "ruleUrl":"http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
  "state":"alerting",
  "tags":{
    "tag name":"tag value"
  }
}

The alert payload

Once an alert is received by Grafana OnCall, the following occurs, based on the alert content:

  1. The most useful information is shown in a readable format.

  2. Noise is minimized by grouping alerts into incidents, combining similar alerts into a single page.

  3. The incident is resolved if the monitoring system tells Grafana OnCall to do so.

In Grafana OnCall every incident and alert has the following fields:

  • Title, message and image url
  • Grouping Id
  • Resolve Signal

The JSON payload is converted. For example:

  • {{ payload.title }} -> Title
  • {{ payload.message }} -> Message
  • {{ payload.imageUrl }} -> Image Url

The result is that each field of the alert in OnCall is now mapped to the JSON payload keys. This also true for the alert behavior:

  • {{ payload.ruleId }} -> Grouping Id
  • {{ 1 if payload.state == 'OK' else 0 }} -> Resolve Signal

OnCall has default Jinja templates for the most popular monitoring systems.

If your monitoring system is not in the Grafana OnCAll integrations list you can create the most generic integration Webhook, send an alert, and write your own templates.

As a best practice, add _Playbooks_, _Useful links_, or _Checklists_ to the alert message.

How to customize templates

You can customize the default templates in Grafana OnCall by opening the Settings window in either the Integrations or Incident tab:

  1. From the Integrations tab, select the integration, then click the Settings (gear) icon.

  2. From the Incidents tab, click Edit rendering, grouping, and other templates

  3. In Settings, select the template to edit from Edit template for.

  4. Edit the Appearances template as needed:

    • Title, Message, Image url for Web
    • Title, Message, Image url for Slack
    • Title used in SMS
    • Title used in Phone
    • Title, Message used in Email
  5. Edit the alert behavior as needed:

    • Grouping Id - This output groups other alerts into a single incident
    • Acknowledge Condition - The output should be ok, true, or 1 to auto-acknowledge the incident. For example, {{ 1 if payload.state == 'OK' else 0 }}.
    • Resolve Condition - The output should be ok, true or 1 to auto-resolve the incident. For example, {{ 1 if payload.state == 'OK' else 0 }}.
    • Source Link - Used to customize the URL link to provide as the “source” of the alert.

Advanced Jinja templates

Grafana OnCall uses Jinja templating language to format incidents in web, Slack, phone calls, SMS messages, and more because the JSON format is not easily readable by humans. As a result, you can decide what you want to see when the incident is triggered as well as how it should be presented.

Jinja2 offers simple but multi-faceted functionality by using loops, conditions, functions, and more.

NOTE: Every alert from a monitoring system comes in the key/value format. Grafana OnCall has rules about which of the keys match to: __title, message, image, grouping, and auto-resolve__.

Loops

Monitoring systems can send an array of values. In this example, you can use Jinja to iterate and format the alert using a Grafana example:

*Values:*
 {% for evalMatch in payload.evalMatches -%}
 `{{ evalMatch['metric'] }}: '{{ evalMatch['value'] -}}'`{{ " " }}
 {%- endfor %}

Conditions

You can add instruction if an alert comes from a specified Grafana alert rule:

{% if  payload.ruleId == '1' -%}
*Alert TODOs*
1. Get acess to the container
    ```
        kubectl port-forward service/example 3000:80
    ```
2. Check for the exception.
3. Open the container and reload caches.
4. Click Custom Button `Send to Jira`
{%- endif -%}

Built-in Jinja functions

Jinja2 includes built-in functions that can also be used in Grafana OnCall. For example:

{{ payload | tojson_pretty }}

Built-in functions:

  • abs
  • capitalize
  • trim
  • You can see the full list of Jinja built-in functions on github here

Functions added by Grafana OnCall

  • time - current time
  • tojson_pretty - JSON prettified
  • iso8601_to_time - converts time from iso8601 (2015-02-17T18:30:20.000Z) to datetime
  • datetimeformat - converts time from datetime to the given format (%H:%M / %d-%m-%Y by default)