Alerting high availability
The Grafana Alerting system has two main components: a
Scheduler and an internal
Scheduler evaluates your alert rules, while the internal Alertmanager manages routing and grouping.
When running Grafana Alerting in high availability, the operational mode of the scheduler remains unaffected, and each Grafana instance evaluates all alerts. The operational change happens in the Alertmanager when it deduplicates alert notifications across Grafana instances.
The coordination between Grafana instances happens via a Gossip protocol. Alerts are not gossiped between instances and each scheduler delivers the same volume of alerts to each Alertmanager.
The two types of messages gossiped between Grafana instances are:
- Notification logs: Who (which instance) notified what (which alert).
- Silences: If an alert should fire or not.
The notification logs and silences are persisted in the database periodically and during a graceful Grafana shut down.