Configure Grafana OnCall on Grafana Labs

Escalation chains and routes

Tue, 07 Apr 2026 10:28:26 +0000

Escalation Chains and Routes

Often alerts from monitoring systems need to be sent to different escalation chains and messaging channels, based on their severity, or other alert content.

Routes

Routes are used to determine which escalation chain should be used for a specific alert group. A route’s Routing Templates are evaluated for each alert and the first matching route is used to determine the escalation chain and chatops channels.

Example:

trigger escalation chain called Database Critical for alerts with {{ payload.severity == "critical" and payload.service == "database" }} in the payload

create a different route for alerts with the payload {{ "synthetic-monitoring-dev-" in payload.namespace }} and select a escalation chain called Security.

Manage routes

Open Integration page
Click Add route button to create a new route
Click Edit button to edit Routing Template. The routing template must evaluate to True for it to apply
Select channels in Publish to Chatops section

Note: If the Publish to Chatops section doesn’t exist, connect Chatops integrations first. For more information, refer to Notify people.
Select Escalation Chain from the list
If Escalation Chain does not exist, click Add new escalation chain button to create a new one, it will open in a new tab.
Once created, Reload list, and select the new escalation chain
Click Arrow Up and Arrow Down on the right to change the order of routes
Click Three dots and Delete Route to delete the route

Routing based on labels

Note: Labels are currently available only in cloud.

In addition, there is a labels variable available to your routing templates, which contains all of the labels assigned to the Alert Group, as a dict. This allows you to route based on labels (or a mix of labels and/or payload based data):

Example:

{{ labels.foo == "bar" or "hello" in labels.keys() or payload.severity == "critical" }}

Escalation Chains

Once an alert group is created and assigned to the route with escalation chain, the escalation chain will be executed. Until user performs an action, which stops the escalation chain (e.g. acknowledge, resolve, silence etc), the escalation chain will continue to execute.

Users can create escalation chains to configure different type of escalation workflows. For example, you can create a chain that will notify on-call users with high priority, and another chain that will only send a message into a Slack channel.

Escalation chains determine Who and When to notify. How to notify is set by the user, based on their own preferences.

Types of escalation steps

Wait - wait for a specified amount of time before proceeding to the next step. If you need a larger time interval, use multiple wait steps in a row.
Notify users - send a notification to a user or a group of users.
Notify users from on-call schedule - send a notification to a user or a group of users from an on-call schedule.
Notify all users from a team - send a notification to all users in a team.
Resolve incident automatically - resolve the alert group right now with status Resolved automatically.
Notify whole slack channel - send a notification to the users in the slack channel. These users will be notified via the method configured in their user profile.
Notify Slack User Group - send a notification to each member of a slack user group. These users will be notified via the method configured in their user profile.
Trigger outgoing webhook - trigger an outgoing webhook.
Notify users one by one (round robin) - each notification will be sent to a group of users one by one, in sequential order in round robin fashion.
Continue escalation if current time is in range - continue escalation only if current time is in specified range. It will wait for the specfied time to continue escalation. Useful when you want to get escalation only during working hours
Continue escalation if >X alerts per Y minutes (beta) - continue escalation only if it passes some threshold
Repeat escalation from beginning (5 times max) - loop the escalation chain

Note: Both “Notify whole Slack channel” and “Notify Slack User Group” will filter OnCall registered users matching the users in the Slack channel or Slack User Group with their profiles linked to their Slack accounts (ie. users should have linked their Slack and OnCall users). In both cases, the filtered users satisfying the criteria above are notified following their respective notification policies. However, to avoid spamming the Slack channel/thread, users won’t be notified in the alert group Slack thread (this is how the feature is currently implemented) but instead notify them using their other defined options in their respective policies.

Notification types

Each escalation step that notifies a user, does so by triggering their personal notification steps. These are configured in the Grafana OnCall users page (by clicking “View my profile”). It will be executed for each user in the escalation step User can configure two types of personal notification chains:

Default Notifications
Important Notifications

In the escalation step, user can select which type of notification to use. For more information, refer to Notify people.

Manage Escalation Chains

Open Escalation Chains page
Click New escalation chain button to create a new escalation chain
Enter a name and assign it to a team

Note: Name must be unique across organization Note: Alert Groups inherit the team from the Integration, not the Escalation Chain
Click Add escalation step button to add a new step
Click Delete to delete the Escalation Chain, and Edit to edit the name or the team.

Important: Linked Integrations and Routes are displayed in the right panel. Any change in the Escalation Chain will affect all linked Integrations and Routes.

Outgoing Webhooks

Tue, 07 Apr 2026 10:28:26 +0000

Outgoing Webhooks

⚠️ A note about (Legacy) webhooks: Webhooks that were created before version v1.3.11 are marked as (Legacy). Do not worry! They are still connected to their respective escalation chains and will continue to execute as they always have.

The (Legacy) webhook is no longer editable due to changes to the internal representation. If you need to edit it you must use the Make a copy action in the menu and make your changes there. This will create the webhook in the new format. Be sure to change your escalation chains to point to the new copy otherwise it will not be active. The (Legacy) webhook can then be deleted.

Outgoing webhooks are used by Grafana OnCall to send data to a URL in a flexible way. These webhooks can be triggered from a variety of event types and make use of Jinja2 to transform data into the format required at the destination URL. Each outgoing webhook receives contextual data when executed which can be processed by Jinja2 templates to customize the request being sent.

Creating an outgoing webhook

To create an outgoing webhook navigate to Outgoing Webhooks and click + Create. On this screen outgoing webhooks can be viewed, edited and deleted. To create the outgoing webhook click New Outgoing Webhook and then select a preset based on what you want to do. A simple webhook will POST alert group data as a selectable escalation step to the specified url. If you require more customization use the advanced webhook which provides all of the fields described below.

Outgoing webhook fields

The outgoing webhook is defined by the following fields. For more information about template usage see Outgoing webhook templates section.

ID

This field is generated after an outgoing webhook has been created. It is used to reference the responses of other webhooks, see Advanced Usage - Using response data for more details.

Name

Display name of the outgoing webhook.

Required	Template Accepted	Default Value
✔️	❌	Empty

Enabled

Controls whether the outgoing webhook will trigger or is ignored.

Required	Template Accepted	Default Value
✔️	❌	True

Assign to Team

Sets which team owns the outgoing webhook for filtering and visibility. This setting does not restrict outgoing webhook execution to events from the selected team.

Required	Template Accepted	Default Value
❌	❌	Empty

Trigger Type

The type of event that will cause this outgoing webhook to execute. The types of triggers are:

For more details about types of triggers see Event types

Required	Template Accepted	Default Value
✔️	❌	None

HTTP Method

The HTTP method used in the request made by the outgoing webhook. This should match what is required by the URL you are sending to.

Required	Template Accepted	Default Value
✔️	❌	POST

Integrations

Restricts the outgoing webhook to only trigger if the event came from a selected integration. If no integrations are selected the outgoing webhook will trigger for any integration.

Required	Template Accepted	Default Value
❌	❌	None

Webhook URL

The destination URL the outgoing webhook will make a request to. This must be a FQDN.

⚠️ Note the destination server must respond back within 4 seconds or it will result in a timeout (this can be seen in the “Response Body” under the “Last Run” section)

Required	Template Accepted	Default Value
✔️	✔️	Empty

Webhook Headers

Headers to add to the outgoing webhook request.

Required	Template Accepted	Default Value
❌	✔️	Empty

Username

Username to use when making the outgoing webhook request.

Required	Template Accepted	Default Value
❌	❌	Empty

Password

Password to use when making the outgoing webhook request.

Required	Template Accepted	Default Value
❌	❌	Empty

Authorization Header

Authorization header to use when making the outgoing webhook request.

Required	Template Accepted	Default Value
❌	❌	None

Trigger Template

A template used to dynamically determine whether the webhook should execute based on the content of the payload. If the template evaluates to Empty, True or 1 the webhook will execute.

Required	Template Accepted	Default Value
❌	✔️	Empty

Data

The main body of the request to be sent by the outgoing webhook.

Required	Template Accepted	Default Value
❌	✔️	Empty

Forward All

Toggle to send the entire webhook payload instead of using the values in the Data field

Required	Template Accepted	Default Value
❌	❌	False

Outgoing webhook templates

The fields that accept a Jinja2 template in outgoing webhooks are able to process data to customize the output. The following is an example of the data available to access from a template. Some data depending on the timing of the webhook and the triggering event may not always be available, see field descriptions specific details. The format you use to call the variables must match the structure of how the fields are nested in the data.

{
  "event": {
    "type": "resolve",
    "time": "2023-04-19T21:59:21.714058+00:00"
  },
  "user": {
    "id": "UVMX6YI9VY9PV",
    "username": "admin",
    "email": "admin@localhost"
  },
  "alert_group": {
    "id": "I6HNZGUFG4K11",
    "integration_id": "CZ7URAT4V3QF2",
    "route_id": "RKHXJKVZYYVST",
    "alerts_count": 1,
    "state": "resolved",
    "created_at": "2023-04-19T21:53:48.231148Z",
    "resolved_at": "2023-04-19T21:59:21.714058Z",
    "acknowledged_at": "2023-04-19T21:54:39.029347Z",
    "title": "Incident",
    "permalinks": {
      "slack": null,
      "telegram": null,
      "web": "https://**********.grafana.net/a/grafana-oncall-app/alert-groups/I6HNZGUFG4K11"
    },
    "labels": {
      "urgency": "3"
    }
  },
  "alert_group_id": "I6HNZGUFG4K11",
  "alert_payload": {
    "endsAt": "0001-01-01T00:00:00Z",
    "labels": {
      "region": "eu-1",
      "alertname": "TestAlert"
    },
    "status": "firing",
    "startsAt": "2018-12-25T15:47:47.377363608Z",
    "annotations": {
      "description": "This alert was sent by user for the demonstration purposes"
    },
    "generatorURL": ""
  },
  "integration": {
    "id": "CZ7URAT4V3QF2",
    "type": "webhook",
    "name": "Main Integration - Webhook",
    "team": "Webhooks Demo",
    "labels": {
      "urgency": "3"
    }
  },
  "alert_group_acknowledged_by": {
    "id": "UVMX6YI9VY9PV",
    "username": "admin",
    "email": "admin@localhost"
  },
  "alert_group_resolved_by": {
    "id": "UVMX6YI9VY9PV",
    "username": "admin",
    "email": "admin@localhost"
  },
  "notified_users": [],
  "users_to_be_notified": [],
  "responses": {
    "WHP936BM1GPVHQ": {
      "id": "7Qw7TbPmzppRnhLvK3AdkQ",
      "created_at": "15:53:50",
      "status": "new",
      "content": {
        "message": "Ticket created!",
        "region": "eu"
      }
    }
  },
  "webhook": {
    "id": "WH9NSKXWPXSNY3",
    "name": "Demo Webhook",
    "labels": {}
  }
}

Outgoing webhook data fields

`event`

Context information about the event that triggered the outgoing webhook.

{{ event.type }} - Lower case string matching type of event
{{ event.time }} - Time event was triggered

`user`

Information about the user if the source of the event was a user. If a user acknowledges an alert group after receiving a notification this field will have that user’s information. If an alert group was auto-resolved based on criteria in the integration this field will be empty.

{{ user.id }} - UID of the user within Grafana OnCall
{{ user.username }} - Username in Grafana
{{ user.email }} - Email associated with user’s Grafana account

`alert_group`

Details about the alert group associated with this event.

{{ alert_group.id }} - UID of alert group
{{ alert_group.integration_id }} - UID of integration that alert came through
{{ alert_group.route_id }} - UID of route of integration that alert came through
{{ alert_group.alerts_count }} - Count of alerts in alert group
{{ alert_group.state }} - Current state of alert group
{{ alert_group.created_at }} - Timestamp alert group was created
{{ alert_group.resolved_at }} - Timestamp alert group was resolved (None if not resolved yet)
{{ alert_group.acknowledged_at }} - Timestamp alert group was acknowledged (None if not acknowledged yet)
{{ alert_group.title }} - Title of alert group
{{ alert_group.permalinks }} - Links to alert group in web and chat ops if available
{{ alert_group.labels }} - Alert group labels

`{{ alert_group_id }}`

UID of alert group, same as {{ alert_group.id }} (For convenience and compatibility with earler versions of Grafana OnCall)

`alert_payload`

Content of the first alert in the alert group. Content will depend on what the alert source has sent. Some commonly used fields are:

{{ alert_payload.labels.alertname }}
{{ alert_payload.annotations.description }}

`integration`

Details about the integration that received this alert

{{ integration.id }} - UID of integration
{{ integration.type }} - Type of integration (grafana, alertmanager, webhook, etc.)
{{ integration.name }} - Name of integration
{{ integration.team }} - Team integration belongs to if integration is assigned to a team
{{ integration.labels }} - Integration labels

`notified_users`

Array of users that have received notifications about the associated alert group. Each user element in the array consists of id,username,email. Depending on timing of events and notifications this might not be populated yet if notifications are still in progress. Access as {{ notified_users[0].username }} for example.

`users_to_notify`

Array of users that could potentially be notified based on the configured escalation chain. Each user element in the array consists of id,username,email. Array elements are ordered based on the order users will be notified with the first element being the user that will be notified next. Like notified_users depending on timing of notifications a user in this array may have already been notified by the time this data is being processed. Access as {{ users_to_notify[0].username }} for example.

`alert_group_acknowledged_by`

Information about the user who acknowledged the alert group

{{ user.id }} - UID of the user within Grafana OnCall
{{ user.username }} - Username in Grafana
{{ user.email }} - Email associated with user’s Grafana account

`alert_group_resolved_by`

Information about the user who resolved the alert group

{{ user.id }} - UID of the user within Grafana OnCall
{{ user.username }} - Username in Grafana
{{ user.email }} - Email associated with user’s Grafana account

`responses`

The responses field is used to access the response data of other webhooks that are associated with this alert group. The keys inside responses are the UID of other outgoing webhooks. The values inside each response is the latest response of the referenced webhook when it was executed on behalf of the current alert group. See Advanced Usage - Using response data for more details. Access as {{ responses["WHP936BM1GPVHQ"].content.message }} for example

`webhook`

Triggered webhook details

{{ webhook.id }} - UID of webhook
{{ webhook.name }} - Name of webhook
{{ webhook.labels }} - Webhook labels

UID

Templates often use UIDs to make decisions about what actions to take if you need to find the UID of an object in the user interface to reference they can be found in the following places:

Outgoing Webhook - In the table there is an info icon, UID displayed on hover, click to copy to clipboard
Integration - In integrations beside the name is an info icon, UID displayed on hover, click to copy to clipboard
Routes - With an integration selected beside Send Demo Alert is an infor icon, UID displayed on hover, click to copy to clipboard
Alert group - When viewing an alert group UID is visible in the browser URL
User - When viewing a user’s profile UID is visible in the browser URL

UIDs are also visible in the browser URL when a specific object is selected for view or edit.

Template examples

The following is an example of an entry in the Data field that would return an alert name and description.

{
  "name": "{{ alert_payload.labels.alertname }}",
  "message": "{{ alert_payload.annotations.description }}"
}

Here is an example using the user’s email address as part of a URL:

https://someticketsystem.com/new-ticket?assign-user={{ user.email }}

Note about JSON

Take this template for example:

{
  "labels": "{{ alert_payload.labels }}"
}

It will result in the following (Invalid JSON due to single quotes):

{
  "labels": {'region': 'eu-1', 'alertname': 'TestAlert'}
}

To fix change the template to:

{
  "labels": {{ alert_payload.labels | tojson()}}
}

Now the result is correct:

{
  "labels": {
    "alertname": "TestAlert",
    "region": "eu-1"
  }
}

Event types

Escalation Step

event.type escalation

This event will trigger when the outgoing webhook is included as a step in an escalation chain.

Alert Group Created

event.type alert group created

This event will trigger when a new alert group is created.

Acknowledged

event.type acknowledge

This event will trigger when a user acknowledges an alert group or an alert group is auto-acknowledged by the integration.

Resolved

event.type resolve

This event will trigger when a user resolves an alert group or an alert group is auto-resolved by the integration.

Silenced

event.type silence

This event will trigger when a user silences an alert group.

Unsilenced

event.type unsilence

This event will trigger when a user unsilences an alert group or a silence expires.

Unresolved

event.type unresolve

This event will trigger when a user unresolves an alert group.

Unacknowledged

event.type unacknowledge

This event will trigger when a user unacknowledges an alert group.

Status Change

event.type status change

This event will trigger when any of the status change actions happen (acknowledge, resolve, silence, unacknowledge, unresolve, or unsilence). The event details included in the payload will match those of the original action triggering the event.

Viewing status of outgoing webhooks

In the outgoing webhooks table if a webhook is enabled Last Run will have the following information:

Timestamp outgoing webhook was triggered
HTTP response code

If more information is required you can click Status in the table. The status drawer shows the following:

Webhook Name
Webhook UID
Trigger Type
Last Run Time
URL
Response Code
Response Body
Trigger Template
Request Headers
Request Data

In the status drawer if a field makes use of a template it will display both the template and the result otherwise it will only display the value. Fields which are not used are not shown.

Advanced usage

Using trigger template field

The trigger template field can be used to provide control over whether a webhook will execute. This is useful in situations where many different kinds of alerts are going to the same integration but only some of them should call the webhook. To accomplish this the trigger template field can contain a template that will process data from the alert group and evaluate to empty, True or 1 if the webhook should execute, any other values will result in the webhook not executing.

Using response data

The responses section of the payload makes available the responses of other webhooks that have acted on the same alert group. To access them responses uses the id of the webhook as a key. The id can be found by hovering over the info icon, clicking will copy the id to the clipboard. The response data of the most recent execution of the webhook for this same alert group can be accessed this way.

The typical application of this is where a webhook will create a ticket in another system and OnCall needs to use the id of that ticket to keep its status synchronized with the state changes being made in OnCall.

Advanced examples

Integrate with third-party services:

Jinja2 templating

Tue, 07 Apr 2026 10:28:26 +0000

Jinja2 templating

Grafana OnCall can integrate with any monitoring system that can send alerts via webhooks with JSON payloads. By default, webhooks deliver raw JSON payloads. When Grafana OnCall receives an alert and parses its payload, a default pre-configured alert template is applied to modify the alert payload to be more human-readable. These alert templates are customizable for any integration. Templates are also used to notify different escalation chains based on the content of the alert payload.

Alert payload

Alerts received by Grafana OnCall contain metadata as keys and values in a JSON object. The following is an example of an alert which was initiated by Grafana Alerting, and received by Grafana OnCall:

{
  "dashboardId": 1,
  "title": "[Alerting] Panel Title alert",
  "message": "Notification Message",
  "evalMatches": [
    {
      "value": 1,
      "metric": "Count",
      "tags": {}
    }
  ],
  "imageUrl": "https://grafana.com/static/assets/img/blog/mixed_styles.png",
  "orgId": 1,
  "panelId": 2,
  "ruleId": 1,
  "ruleName": "Panel Title alert",
  "ruleUrl": "http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
  "state": "alerting",
  "tags": {
    "tag name": "tag value"
  }
}

In Grafana OnCall every alert and alert group have the following fields:

Title, Message and Image Url for each notification method (Web, Slack, Ms Teams, SMS, Phone, Email, etc.)
Grouping Id - unique identifier for each non-resolved alert group
Resolved by source
Acknowledged by source
Source link

The JSON payload is converted to OnCall fields. For example:

{{ payload.title }} -> Title
{{ payload.message }} -> Message
{{ payload.imageUrl }} -> Image Url

The result is that each field of the alert in OnCall is now mapped to the JSON payload keys. This also true for the alert behavior:

{{ payload.ruleId }} -> Grouping Id
{{ 1 if payload.state == 'OK' else 0 }} -> Resolve Signal

Grafana OnCall provides pre-configured default Jinja templates for supported integrations. If your monitoring system is not in the Grafana OnCall integrations list, you can create a generic webhook integration, send an alert, and configure your templates.

Types of templates

Alert templates allow you to format any alert fields recognized by Grafana OnCall. You can customize default alert templates for all the different notification methods. For more advanced customization, use Jinja templates.

Routing template

Routing Template - used to route alerts to different Escalation Chains based on alert content (conditional template, output should be True)

Note: For conditional templates, the output should be True to be applied, for example {{ True if payload.state == 'OK' else False }}

Appearance templates

How alerts are displayed in the UI, messengers, and notifications

Title, Message, Image url for Web
Title, Message, Image url for Slack
Title, Message, Image url for MS Teams
Title, Message, Image url for Telegram
Title for SMS
Title for Phone Call
Title, Message for Email
Title, Message for Mobile app push notifications

Behavioral templates

Grouping Id - applied to every incoming alert payload after the Routing Template. It can be based on time, alert content, or both. If the resulting grouping id matches an existing non-resolved alert group grouping id, the alert will be grouped accordingly. Otherwise, a new alert group will be created
Autoresolution - used to auto-resolve alert groups with status Resolved by source (Conditional template, output should be True)
Auto acknowledge - used to auto-acknowledge alert groups with status Acknowledged by source (Conditional template, output should be True)
Source link - Used to customize the URL link to provide as the “source” of the alert.

Note: For conditional templates, the output should be True to be applied, for example {{ True if payload.state == 'OK' else False }}

Pro Tip: As a best practice, add Playbooks, Useful links, or Checklists to the alert message.

How to edit templates

Open the Integration page for the integration you want to edit 1`. Click the Edit button for the Templates Section. Now you can see previews of all templates for the Integration
Select the template you want to edit and click the Edit button to the right to the template name. The template editor will open. The first column is the example alert payload, second column is the Template itself, and third column is used to view rendered result.
Select one of the Recent Alert groups for the integration to see its latest alert payload. If you want to edit this payload, click the Edit button right to the Alert Group Name.
Alternatively, you can click Use custom payload and write your own payload to see how it will be rendered
Press Control + Enter in the editor to see suggestions
Click Cheatsheet in the second column to get some inspiration.
If you edit Messenger templates, click Save and open Alert Group in ChatOps to see how the alert will be rendered in the messenger, right in the messenger (Only works for an Alert Group that exists in the messenger)
Click Save to save the template

Advanced Jinja templates

Grafana OnCall uses the Jinja templating language to format alert groups for the Web, Slack, phone calls, SMS messages, and more. As a result, you can decide what you want to see when an alert group is triggered, as well as how it should be presented.

Jinja2 offers simple but multi-faceted functionality by using loops, conditions, functions, and more.

NOTE: Every alert from a monitoring system comes in the key/value format.

Grafana OnCall has rules about which of the keys match to: __title, message, image, grouping, and auto-resolve__.

Loops

Monitoring systems can send an array of values. In this example, you can use Jinja to iterate and format the alert using a Grafana example:

*Values:*
 {% for evalMatch in payload.evalMatches -%}
 `{{ evalMatch['metric'] }}: '{{ evalMatch['value'] -}}'`{{ " " }}
 {%- endfor %}

Conditions

You can add instructions if an alert comes from a specified Grafana alert rule:

{% if  payload.ruleId == '1' -%}
*Alert TODOs*
1. Get acess to the container
    ```
        kubectl port-forward service/example 3000:80
    ```
2. Check for the exception.
3. Open the container and reload caches.
4. Click Custom Button `Send to Jira`
{%- endif -%}

Built-in Jinja functions

Jinja2 includes built-in functions that can also be used in Grafana OnCall. For example:

{{ payload | tojson_pretty }}

Built-in functions:

abs
capitalize
trim
You can see the full list of Jinja built-in functions on github here

Functions added by Grafana OnCall

time - current time
tojson - dumps a structure to JSON
tojson_pretty - same as tojson, but prettified
iso8601_to_time - converts time from iso8601 (2015-02-17T18:30:20.000Z) to datetime
datetimeformat - converts datetime to string according to strftime format codes (%H:%M / %d-%m-%Y by default)
datetimeformat_as_timezone - same as datetimeformat, with the inclusion of timezone conversion (UTC by default)
- Usage example: {{ payload.alerts.startsAt | iso8601_to_time | datetimeformat_as_timezone('%Y-%m-%dT%H:%M:%S%z', 'America/Chicago') }}
datetimeparse - converts string to datetime according to strftime format codes (%H:%M / %d-%m-%Y by default)
regex_replace - performs a regex find and replace
regex_match - performs a regex match, returns True or False
- Usage example: {{ payload.ruleName | regex_match(".*") }}
b64decode - performs a base64 string decode
- Usage example: {{ payload.data | b64decode }}
parse_json - parses a given json string to an object
- Usage example: {{ (payload.data | b64decode | parse_json).name }}