Connect to Grafana OnCall on Grafana Labs

Integrate with data sources

Tue, 18 Mar 2025 12:53:54 +0100

Integrate with data sources

Grafana OnCall can connect directly to the monitoring services of your data sources listed in the Grafana OnCall Integrations section.

Connect to a data source with configured alerts.

In Grafana OnCall, click on the Integrations tab and click + New integration for receiving alerts.
Select an integration from the provided options.

If you want to use an integration that is not listed, you must use webhooks. To learn more about using webhooks see Integrate with webhooks.
Configure your integration.

Each integration has a different method of connecting to Grafana OnCall. For example, if you want to connect to your Grafana data source, select Grafana and follow the instructions.

Configure alert notifications with Grafana Alerting

Wed, 19 Apr 2023 14:16:27 +0100

Connect Grafana Alerting to Grafana OnCall

You must have the Admin role assigned to connect to Grafana OnCall.

Navigate to the Integrations tab in Grafana OnCall.
Click on the Grafana logo.
Follow the instructions that display in the dialog box to find a unique integration URL in the monitoring configuration.

Grafana installations

Grafana OnCall can be set up using two methods:

Grafana Alerting: Grafana OnCall is connected to the same Grafana instance being used to manage Grafana OnCall.
Grafana (External): Grafana OnCall is connected to one or more Grafana instances separate from the one being used to manage Grafana OnCall.

Grafana Cloud Alerting

Use the following method if you are connecting Grafana OnCall with alerts coming from the same Grafana instance from which Grafana OnCall is being managed.

In Grafana OnCall, navigate to the Integrations tab and select New Integration for receiving alerts.
Click Quick connect in the Grafana Alerting tile. This will automatically create the integration in Grafana OnCall as well as the required contact point in Alerting.

Note: You must connect the contact point with a notification policy. For more information, see Contact points in Grafana Alerting
Determine the escalation chain for the new integration by either selecting an existing one or by creating a new chain.
In Grafana Cloud Alerting, navigate to Alerting > Contact Points and find a contact point with a name matching the integration you created in Grafana OnCall.
Click the Edit (pencil) icon, then click Test. This will send an alert to Grafana OnCall.

Grafana (External)

Connect Grafana OnCall with alerts coming from an instance of Grafana different from the one on which Grafana OnCall is being managed:

In Grafana OnCall, navigate to the Integrations tab and select New Integration for receiving alerts.
Select the Grafana tile.
View and save the URL needed to connect.
Determine the escalation chain for the new integration by either selecting an existing one or by creating a new chain.
Go to the other Grafana instance to connect to Grafana OnCall and navigate to Alerting > Contact Points.
Select New Contact Point.
Choose the contact point type webhook, then paste the URL generated in step 3 into the URL field.

Note: You must connect the contact point with a notification policy. For more information, see Contact points in Grafana Alerting.
Click the Edit (pencil) icon, then click Test. This will send an alert to Grafana OnCall.

Configure alert notifications with Alertmanager

Wed, 19 Apr 2023 14:16:27 +0100

Alertmanager (Prometheus)

The Alertmanager integration handles alerts sent by client applications such as the Prometheus server.

Grafana OnCall provides grouping abilities when processing alerts from Alertmanager, including initial deduplicating, grouping, and routing the alerts to Grafana OnCall.

Connect Alertmanager to Grafana OnCall

You must have an Admin role to connect to Grafana OnCall.

Navigate to the Integrations tab in Grafana OnCall.
Click on the Alertmanager icon.
Follow the instructions that display in the dialog box to find a unique integration URL in the monitoring configuration.

Configure Alertmanager

Update the receivers section of your Alertmanager configuration to use a unique integration URL:

route:
  receiver: 'oncall'
  group_by: [alertname, datacenter, app]

receivers:
- name: 'oncall'
  webhook_configs:
  - url: <integation-url>
    send_resolved: true

Configure grouping with Alertmanager and Grafana OnCall

You can use the grouping mechanics of Alertmanager and Grafana OnCall to configure settings for groups of alert notifications.

Alertmanager offers three grouping settings:

group_by provides two options, instance or job.
group_wait sets the length of time to initially wait before sending a notification for a particular group of alerts. For example, group_wait can be set to 45s.

Setting a high value for group_wait reduces alert noise and minimizes interruption, but it may introduce longer delays in receiving alert notifications. To set an appropriate wait time, consider whether the group of alerts will be the same as those previously sent.
group_interval sets the length of time to wait before sending notifications about new alerts that have been added to a group of alerts that have been previously alerted on. This setting is usually set to five minutes or more.

During high alert volume periods, Alertmanager will send alerts at each group_interval, which can mean a lot of distraction. Grafana OnCall grouping will help manage this in the following ways:
- Grafana OnCall groups alerts based on the first label of each alert.
- Grafana OnCall marks an incident as resolved only when the amount of grouped alerts with state resolved equals the amount of alerts with state firing.

Configure alert notifications with Zabbix

Tue, 11 Mar 2025 08:38:14 -0600

Connect Zabbix to Grafana OnCall

Zabbix is an open-source monitoring software tool for diverse IT components, including networks, servers, virtual machines, and cloud services. Zabbix provides monitoring for metrics such as network utilization, CPU load, and disk space consumption.

To connect Grafana OnCall with Zabbix using the following procedure, you must have an Admin role assigned.

Navigate to the Integrations tab in Grafana OnCall.
Follow the instructions that display in the dialog box to use a unique integration URL in the monitoring configuration.

Configure the Zabbix server

Deploy a Zabbix playground if you don’t have one set up:

 docker run --name zabbix-appliance -t \
      -p 10051:10051 \
      -p 80:80 \
      -d zabbix/zabbix-appliance:latest

Establish an ssh connection to a Zabbix server.
Bash
```
docker exec -it zabbix-appliance bash
```
Place the grafana_oncall.sh script in the AlertScriptsPath directory specified within the Zabbix server configuration file (zabbix_server.conf).
Bash
```
grep AlertScriptsPath /etc/zabbix/zabbix_server.conf
```
Note: The script must be executable by the user running the zabbix_server binary (usually “zabbix”) on the Zabbix server. For example, chmod +x grafana_oncall.sh
Bash
```
ls -lh /usr/lib/zabbix/alertscripts/grafana_oncall.sh
-rw-r--r--    1 root     root        1.5K Jun  6 07:52 /usr/lib/zabbix/alertscripts/grafana_oncall.sh
```

Configure Zabbix alerts

Within Zabbix web interface, do the following:

In a browser, open localhost:80.
Navigate to Adminitstration > Media Types > Create Media Type.
Create a Media Type with the following fields.
- Name: Grafana OnCall
- Type: script
- Script parameters:
  - {ALERT.SENDTO}
  - {ALERT.SUBJECT}
  - {ALERT.MESSAGE}

Set the {ALERT.SEND_TO} value

To send alerts to Grafana OnCall, the {ALERT.SEND_TO} value must be set in the user media configuration.

In the web UI, navigate to Administration > Users and open the user properties form.
In the Media tab, click Add and copy the link from Grafana OnCall in the Send to field.
Click Test in the last column to send a test alert to Grafana OnCall.
Specify Send to OnCall using the unique integration URL from the above step in the testing window that opens.
Create a test message with a body and optional subject and click Test.

Grouping and auto-resolve of Zabbix notifications

Grafana OnCall provides grouping and auto-resolve of Zabbix notifications. Use the following procedure to configure grouping and auto-resolve.

Provide a parameter as an identifier for group differentiation to Grafana OnCall.
Append that variable to the subject of the action as ONCALL_GROUP: ID, where ID is any of the Zabbix macros. For example, {EVENT.ID}. The Grafana OnCall script grafana_oncall.sh extracts this event and passes the alert_uid to Grafana OnCall.
To enable auto-resolve within Grafana Oncall, the “Resolved” keyword is required in the Default subject field in Recovered operations.

grafana_oncall.sh script

#!/bin/bash
# This is the modification of original ericos's shell script.

# Get the url ($1), subject ($2), and message ($3)
url="$1"
subject="${2//$'\r\n'/'\n'}"
message="${3//$'\r\n'/'\n'}"

# Alert state depending on the subject indicating whether it is a trigger going in to problem state or recovering
recoversub='^RECOVER(Y|ED)?$|^OK$|^Resolved.*'

if [[ "$subject" =~ $recoversub ]]; then
    state='ok'
else
    state='alerting'
fi

payload='{
    "title": "'${subject}'",
    "state": "'${state}'",
    "message": "'${message}'"
}'

# Alert group identifier from the subject of action. Grouping will not work without ONCALL_GROUP in the action subject
regex='ONCALL_GROUP: ([a-zA-Z0-9_\"]*)'
if [[ "$subject" =~ $regex ]]; then
    alert_uid=${BASH_REMATCH[1]}
    payload='{
        "alert_uid": "'${alert_uid}'",
        "title": "'${subject}'",
        "state": "'${state}'",
        "message": "'${message}'"
    }'
fi

return=$(curl $url -d "${payload}" -H "Content-Type: application/json" -X POST)

More Information

For more information on Zabbix scripts, see scripts for notifications.

Use webhooks to send and receive alerts

Tue, 11 Mar 2025 08:38:14 -0600

Use webhooks to send and receive alerts

You can use webhooks to send alert group notifications, and also to receive alerts in the event that the data source for for the alerts is not directly supported by Grafana OnCall. You can also use custom templates to format your alerts.

Follow these links to learn more about using webhooks for OnCall alert notifications: