Alertmanager integration for Grafana OnCall
The Alertmanager integration for Grafana OnCall handles alerts sent by client applications such as the Prometheus server.
Grafana OnCall provides grouping abilities when processing alerts from Alertmanager, including initial deduplicating, grouping, and routing the alerts to Grafana OnCall.
Configure Alertmanager integration for Grafana OnCall
You must have an Admin role to create integrations in Grafana OnCall.
In the Integrations tab, click + New integration for receiving alerts.
Select Alertmanager from the list of available integrations.
Follow the instructions in the How to connect window to get your unique integration URL and identify next steps.
receivers section of your Alertmanager configuration to use a unique integration URL:
route: receiver: 'oncall' group_by: [alertname, datacenter, app] receivers: - name: 'oncall' webhook_configs: - url: <integation-url> send_resolved: true
Configure grouping with Alertmanager and Grafana OnCall
You can use the alert grouping mechanics of Alertmanager and Grafana OnCall to configure your alert grouping preferences.
Alertmanager offers three alert grouping options:
group_byprovides two options,
group_waitsets the length of time to initially wait before sending a notification for a particular group of alerts. For example,
group_waitcan be set to 45s.
Setting a high value for
group_waitreduces alert noise and minimizes interruption, but it may introduce delays in receiving alert notifications. To set an appropriate wait time, consider whether the group of alerts will be the same as those previously sent.
group_intervalsets the length of time to wait before sending notifications about new alerts that have been added to a group of alerts that have been previously alerted on. This setting is usually set to five minutes or more.
During high alert volume periods, Alertmanager will send alerts at each
group_interval, which can mean a lot of distraction. Grafana OnCall grouping will help manage this in the following ways:
Grafana OnCall groups alerts based on the first label of each alert.
Grafana OnCall marks an incident as resolved only when the amount of grouped alerts with state
resolvedequals the amount of alerts with state
Related Oncall resources
Getting started with Grafana OnCall for on-call management
This webinar goes over initial Grafana OnCall configuration, setting up an on-call management process, grouping, alert rendering, and organizing runbooks.
Unified alerting and on-call management in Grafana
In this webinar, we’ll demo the Grafana 8 unified alerting system, and introduce Grafana OnCall, our on-call management tool to resolve incidents faster.
Reducing resolution time (MTTR) with the Grafana Stack: Metrics, logs, tracing, & on-call management
See a live demo of Grafana OnCall, and how to correlate your metrics, logs, and traces to reduce you /MTTR using Grafana’s “single pane of glass” approach.