Alerts

Alerts

There are two options for alerting in Grafana Cloud: standard Grafana alerting or Prometheus-style alerting rules. Although they share some common functionality, like the ability to integrate with external sources, they are two different systems that address different use cases. It is possible to use both methods in conjunction with each other.

Prometheus alerting rules

Prometheus-style alerts are built by querying directly from the data source itself. Because these alerts are based on the data, they are not tied to a single panel, making it possible to evaluate and centrally manage alerts across several different Prometheus and Loki data sources.

Differences

  • Alerts can be centrally managed and created across many systems, teams, and dashboards and not bound to just one.
  • Alerts can be created for both metric and log data, based on Prometheus and Loki, respectively.
  • Alerts are not limited to just the graph panel.
  • Silencing and mutes can be created and scheduled in bulk for many alerts using the Alertmanager.
  • Alert notifications can be routed to many external notifier systems using Alertmanager configurations
  • Alert notifications are automatically deduped.
  • Alerts can only be edited Admin roles

Configuration methods

In a traditional on-prem environment, the alert configuration is done through the combination of defining a Prometheus configuration file and an Alertmanager configuration file, which live close to the Prometheus server. With Grafana Cloud, this setup is still possible, as well as additional new ones.

  • You can use the cortextool to upload your configuration files to be hosted and entirely evaluated in Grafana Cloud.
  • You can manage both alerting rules and Alertmanager configurations directly through the Grafana Cloud Alerting interface. Configuration files are unnecessary with this setup.
  • You can use both methods with one another to manage the alerts. For example, updates made using the cortextool are automatically updated and visible within the Grafana Cloud Alerting interface.

TIP: Grafana Cloud Alerting is built off of Prometheus-style alerts and works really well with our agent!

Grafana alerting

Grafana alerts are dashboard-driven and can only be created for the Graph panel. This style of alerting builds on top of the query defined for the graph visualization, so alerts and notifications are sent based on breaking some threshold in the associated panel.

This also means that there is a one-to-one relationship between a Grafana alert and a graph panel, so although Grafana alerts can be viewed centrally, they can only be managed directly from the panel that they’re tied to. As a result, Grafana alerting, is best suited for smaller setups, where there are only a few individuals or teams responsible for a small set of dashboards, where there is little dependencies between them.

Differences

  • Alerts are visual, with an associated alerting threshold line
  • Alerts work with many non-Prometheus data sources, including Graphite
  • Alert notifications can be routed to many external notifier systems, directly from Grafana
  • Alerts are directly associated with a dashboard
  • Alerts can be tested
  • Alerts can be edited by both Editor and Admin roles