Documentation Plugins Databricks data source Alerting

Databricks alerting

The Databricks data source supports Grafana alerting. You can create alert rules from Databricks SQL queries to monitor your data and notify your team when conditions are met. For an overview of alerting, refer to Grafana Alerting.

Supported authentication methods for alerting

Alert rules run on a schedule in the background, without a logged-in user present. Authentication methods that depend on an individual user’s identity can’t run alert rules.

Create an alert rule

To create an alert rule from a Databricks query:

1. Write and test your query in a dashboard panel or in Explore.
2. Make sure the query returns a numeric value that an alert condition can evaluate.
3. Create the alert rule. Refer to Configure alert rules for the full workflow.

An alert query should reduce to a single numeric value that the alert condition can compare against a threshold. For example, the following query counts error events in the evaluation window:

SQL Copy

SELECT count(*) AS error_count
FROM main.logs.application_events
WHERE severity = 'ERROR'
  AND $__timeFilter(event_time)

You can then add a threshold condition, such as error_count IS ABOVE 0, to fire the alert. If your query returns multiple rows or columns, use a reduce expression to collapse the result to a single value before the threshold condition.

Best practices for alert queries

Alert rules evaluate on a schedule, so an inefficient query runs repeatedly and adds load to your SQL warehouse. Follow these practices to keep alerting reliable:

• Filter to a bounded time range. Use the $__timeFilter macro or an explicit WHERE clause so each evaluation scans only recent data instead of the whole table. Refer to Macros.
• Return a small result. Aggregate to a single numeric value rather than selecting many rows. Avoid SELECT *.
• Choose a sensible evaluation interval. Frequent evaluation of heavy queries multiplies warehouse load and cost.

Retry and timeout settings

The retry and timeout settings you configure on the data source also apply to alert rule evaluations:

• Timeout - If a Databricks SQL warehouse is idle, the first query can be slow while the warehouse starts. Set the Timeout high enough that scheduled evaluations don’t fail while the warehouse resumes. Consider a serverless SQL warehouse for faster startup.
• Retries and Retry timeout - Alert evaluations are retried on transient errors using the same retry settings as dashboard queries. Refer to Troubleshooting for the list of errors that trigger a retry.

Handle No Data and Error states

A Databricks SQL warehouse that has scaled to zero can take time to resume, so the first scheduled evaluation after an idle period might time out and put the rule in the Error state. Plan for this:

• Configure how the rule behaves when it returns no data or encounters an error. Refer to Configure no data and error handling.
• Set the Timeout and evaluation interval high enough to tolerate warehouse startup, or use a serverless SQL warehouse for faster resume times.

For more detail, refer to Alert evaluation times out while the SQL warehouse starts.