Grafana Cloud Alerting
Grafana Cloud Alerting allows you to create and manage all of your Prometheus-style alerting rules, for both Prometheus metrics and Loki log data. With this feature, you don’t need to leave Grafana, upload or edit configuration files, or install additional tools.
All members of an organization that have alerts set up can view alerts in Grafana Cloud Alerting. This includes everyone with a Viewer, Editor, or Admin role.
Users with the organization Admin role can also create, edit, or delete alerts.
Grafana Cloud Alerting supports rule management across multiple data sources, for both metrics and logs, across all of the stacks in your org. If you have more than one Prometheus or Loki data source, there will be a dropdown at the top for you to select the data source to configure rules.
NOTE: Pay attention to which data source you select. Cloud alerts are tied to a specific data source. For example, if you have a Loki data source selected you will not be able to create an alert based on a Prometheus data source.
Alerts and recording rules
Prometheus supports two types of rules:
- Recording rules - Recording rules allow you to execute expressions or queries, by saving them off as a stored rule instead.
- Alerting rules - Alerting rules allow you to define alert conditions and to route those notifications to an external service. An alert fires if metrics meet criteria defined in the alerting rule.
Both of these rules are configurable from the Grafana Cloud Alerting interface and configured in the same way.
Alert states are identical to the standard format found in Prometheus rule configurations. In Grafana Cloud Alerting, each individual alert is highlighted by its state to more clearly distinguish between alerts.
- Firing - Alerts that have been active for longer than the configured threshold. Alerts are highlighted in red and tagged with a red
- Pending - Alerts that have been active for less than the configured threshold. Alerts are highlighted in orange.
- Inactive - Alerts that are neither firing nor pending. Alerts are highlighted in green.
The Notifications tab is where you can view all current notifications and sort them by various states, receivers, and labels.
There is a limit on how many rules can be created in a rule group. There is also a limit on how many rule groups can be created.
You can create:
- 15 rules per rule group
- 20 rule groups
If you exceed the limits, you will encounter an error similar to this:
ERRO requests failed fields.msg="request failed with response body per-user rules per rule group limit (limit: 20 actual: 22) exceeded\n" status="400 Bad Request" ERRO unable to load rule group error="failed request to the cortex api" group=limit_rules_per_group namespace=test
The solution is to remove any rules or rule groups in excess of the limits above. Alternately, you can contact support to upgrade your account.