OpenTSDB alerting
You can use Grafana Alerting with OpenTSDB to create alerts based on your time series data. This allows you to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.
For general information about Grafana Alerting, refer to Grafana Alerting.
Before you begin
Before creating alerts with OpenTSDB, ensure you have:
- An OpenTSDB data source configured in Grafana. Refer to Configure the OpenTSDB data source.
- Appropriate permissions to create alert rules.
- Understanding of the metrics you want to monitor.
Supported features
OpenTSDB alerting works with standard metric queries that return time series data. The following table summarizes alerting compatibility:
Create an alert rule
To create an alert rule using OpenTSDB:
- Navigate to Alerting > Alert rules.
- Click New alert rule.
- Enter a name for the alert rule.
- Select your OpenTSDB data source.
- Build your query:
- Select the metric to monitor.
- Choose an appropriate aggregator (for example,
avg,sum,max). - Add tag filters to target specific resources.
- Enable downsampling with an interval matching your evaluation frequency.
- Configure the alert condition (for example, when the value is above a threshold).
- Set the evaluation interval and pending period.
- Configure notifications and labels.
- Click Save rule.
For detailed instructions, refer to Create a Grafana-managed alert rule.
Example alert queries
The following examples show common alerting scenarios with OpenTSDB.
Alert on high CPU usage
Monitor CPU usage and alert when it exceeds 90%:
Condition: When average is above 90
Alert on low disk space
Monitor available disk space and alert when it drops below a threshold:
Condition: When minimum is below 10737418240 (10 GB in bytes)
Alert on high network traffic rate
Monitor network bytes received and alert on high traffic:
Condition: When sum is above 104857600 (100 MB/s in bytes)
Alert on error count spike
Monitor application error counts:
Condition: When sum is above 100
Limitations
When using OpenTSDB with Grafana Alerting, be aware of the following limitations.
Template variables not supported
Alert queries can’t contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like $hostname or $environment aren’t resolved.
If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.
Query complexity
Complex queries with many tags or long time ranges may timeout or fail to evaluate. Simplify queries for alerting by:
- Using specific tag filters instead of wildcards where possible.
- Enabling downsampling with appropriate intervals.
- Reducing the evaluation time range.
OpenTSDB 2.4 behavior
When using OpenTSDB 2.4 with alerting, Grafana executes queries with the parameter arrays=true. This causes OpenTSDB to return data points as an array of arrays instead of a map of key-value pairs. Grafana automatically converts this data to the appropriate format.
Best practices
Follow these best practices when creating OpenTSDB alerts:
- Use specific tag filters: Add tag filters to focus on relevant resources and improve query performance.
- Match downsample interval to evaluation: Set the downsample interval to match or be slightly smaller than your alert evaluation interval.
- Test queries first: Verify your query returns expected results in Explore before creating an alert.
- Set realistic thresholds: Base alert thresholds on historical data patterns to avoid false positives.
- Use meaningful names: Give alert rules descriptive names that indicate what they monitor.
- Enable downsampling: Always enable downsampling for alerting queries to reduce data volume and improve reliability.
- Consider counter resets: For counter metrics, enable the Counter option and set appropriate max values to handle resets correctly.
Next steps
- Build queries to explore your metrics before creating alerts.
- Troubleshoot issues if alerts aren’t firing as expected.
