Alerts

Alerts

There are two options for alerting in Grafana Cloud: standard Grafana alerting or Prometheus-style alerting rules. Although they share some common functionality, like the ability to integrate with external sources, they are two different systems that address different use cases. It is possible to use both methods in conjunction with each other.

Prometheus alerting rules

Prometheus-style alerts are built by querying directly from the data source itself. Because these alerts are based on the data, they are not tied to a single panel. This makes it possible to evaluate and centrally manage alerts across several different Prometheus and Loki data source instances.

Differences

In Grafana Cloud, alerts are not limited to just the graph panel. In addition, you can:

  • Centrally manage and create alerts across many systems, teams, and dashboards. Alerts are not bound to just one.
  • Create alerts for both metric and log data, based on Prometheus and Loki, respectively.
  • Silence and mute alerts in bulk, even using a schedule, using the Alertmanager.
  • Route alert notifications to many external notifier systems using Alertmanager configurations
  • Dedupe alert notifications automatically.
  • Prevent alerts from being edited, except by users with accounts that are assigned Admin roles.

Configuration methods

In a traditional on-prem environment, the alert configuration is done through the combination of defining a Prometheus configuration file and an Alertmanager configuration file, which live close to the Prometheus server. With Grafana Cloud, this setup is still possible, as well as additional new ones.

  • You can use the cortextool to upload your configuration files to be hosted and evaluated entirely in Grafana Cloud.
  • You can manage both alerting rules and Alertmanager configurations directly through the Grafana Cloud Alerting interface. Configuration files are unnecessary with this setup.
  • You can use both methods concurrently to manage the alerts. For example, updates made using the cortextool are automatically updated and visible within the Grafana Cloud Alerting interface.

Tip: Grafana Cloud Alerting is built off of Prometheus-style alerts and works really well with our agent!

Grafana alerting

Grafana alerts are dashboard-driven and can only be created for the Graph panel. This style of alerting builds on top of the query defined for the graph visualization, so alerts and notifications are sent based on breaking some threshold in the associated panel.

This also means that there is a one-to-one relationship between a Grafana alert and a graph panel. So although Grafana alerts can be viewed centrally, they can only be managed directly from the panel that they’re tied to. As a result, Grafana alerting is best suited for smaller setups, where there are only a few individuals or teams responsible for a small set of dashboards and where there are few dependencies between them.

Differences

  • Alerts are visual, with an associated alerting threshold line
  • Alerts work with many non-Prometheus data sources, including Graphite
  • Alert notifications can be routed to many external notifier systems, directly from Grafana
  • Alerts are directly associated with a dashboard
  • Alerts can be tested
  • Alerts can be edited by both Editor and Admin roles