Documentation Grafana Cloud Observe and act Test and validate Synthetic Monitoring Configure alerts Configure legacy alerts

Configure legacy alerts

Legacy alerts are data source-managed alert rules that integrate with Grafana Alerting to trigger alerts based on the pass/fail result of a check run. They use low, medium, and high sensitivity thresholds that are shared across all checks.

Synthetic Monitoring includes three preconfigured legacy alert rules. Each rule fires when a check’s success rate remains below its threshold for five minutes:

• HighSensitivity: Success rate is below 95%.
• MedSensitivity: Success rate is below 90%.
• LowSensitivity: Success rate is below 75%.

Create the legacy alert rules

To create the legacy alert rules:

1. Navigate to Testing & synthetics > Synthetics > Alerts (Legacy).
2. If you haven’t already set up the legacy alert rules, click Populate default alerts. Although the button refers to default alerts, it creates the legacy alert rules described on this page.
3. The HighSensitivity, MedSensitivity, and LowSensitivity rules are automatically generated. These rules query the probe success percentage and check options to decide whether to fire alerts.

You only have to do this once for your Grafana stack.

Enable legacy alerts on a check

To configure a check to trigger an alert:

1. Navigate to Testing & synthetics > Synthetics > Checks.

2. Click Create new check to create a new check or edit a preexisting check in the list.

3. Click the Alerting section to show the alerting fields.

4. Select a sensitivity level to associate with the check and click Save.

   This sensitivity value is published to the alert_sensitivity label on the sm_check_info metric each time the check runs on a probe. The legacy alerts use that label value to determine which checks to fire alerts for.

Checks that have enabled a sensitivity level trigger their corresponding alerts when the success percentage drops below their thresholds.

Configuring the alert sensitivity option sets the sensitivity metric label value that determines which legacy alert rule applies to the check.

How legacy alert rules work

The legacy alert rules use one recording rule and three alert rules, one for each sensitivity level: high, medium, and low. A recording rule runs a query on a schedule and saves the result as a new time series. This lets the legacy alert rules reuse the calculated five-minute success rate instead of recalculating it separately for each sensitivity level.

The recording rule (instance_job_severity:probe_success:mean5m) queries the Prometheus check metrics and calculates the success rate for checks that have an alert_sensitivity label. It saves the results as the following precomputed metrics:

• instance_job_severity:probe_success:mean5m{alert_sensitivity="high"}
• instance_job_severity:probe_success:mean5m{alert_sensitivity="medium"}
• instance_job_severity:probe_success:mean5m{alert_sensitivity="low"}

Then, each legacy alert rule queries its corresponding metric and evaluates it against its threshold to decide whether to fire the alert.

For example, if a check has the alert_sensitivity=high, its success rate is evaluated and compared to its threshold (which defaults to 95%). If the success rate drops below the threshold, the alert rule enters a pending state. When the success rate remains below the threshold for the duration of the pending state (defaults to 5m), the rule starts firing. For further details, refer to Alert Rule Evaluation.

You can edit the threshold values and the duration of the pending state, but you can’t edit the predefined alert_sensitivity values.

Avoid alert-flapping

When enabling alerting for a check, it’s recommended to run that check from multiple locations, preferably three or more. That way, if there’s a problem with a single probe or the network connectivity from that single location, you won’t be needlessly alerted, as the other locations running the same check will continue to report their results alongside the problematic location.