Common mistakes to avoid
These issues appear frequently in production alerting setups:
| Issue | Mistake | Fix |
|---|---|---|
| Hard to identify the problem from the notification | Vague alert names | Use names that describe what and where |
| Outages go unnoticed when data stops arriving | No-data set to Normal | Use Keep Last State or Alerting for critical rules |
| Alerts fire on temporary spikes | Missing pending period | Set a pending period appropriate for your data |
| Incorrect values from counter metrics | Skipping a range function on a counter like quickpizza_server_http_requests_total | Add Increase (or Rate) over a time window before aggregating |
| The alert fires on data that hasn’t fully arrived yet | Checking incomplete data | Adjust the query time range to exclude the most recent, incomplete data |
| The alert has nothing to evaluate | Skipping the query preview | Run Preview and confirm one or more time series appear before saving |
Script
Here are the most common mistakes seen in production alerting setups. These come from real support patterns.
Vague alert names make it hard to know what’s wrong when a notification arrives. Descriptive names like QuickPizza server errors, or on production data, Payment service error rate above five percent, pay off immediately.
Setting no data handling to Normal sounds safe, but it means outages where the data source stops reporting go completely unnoticed. Use Keep Last State or Alerting for critical rules.
Missing a pending period means your alert fires on every temporary spike and resolves a minute later. That creates noise and trains your team to ignore alerts.
And querying a counter without a range function like Increase produces a monotonically rising number that always exceeds a threshold of zero. Always convert counters into a windowed value before you alert.