Slide 4 of 6

Anatomy of an alert rule in Terraform

Alerting resources

Grafana alerting resources in Terraform include:

Terraform resourceWhat it managesKey attributes
grafana_rule_groupA group of alert rules with shared evaluation intervalname, folder_uid, interval_seconds, rule blocks
grafana_contact_pointNotification destinations (email, Slack, PagerDuty, webhooks)name, integration-specific blocks (email, slack, etc.)
grafana_notification_policyRouting rules that match alerts to contact pointscontact_point, group_by, matcher blocks
grafana_mute_timingTime windows when notifications are suppressedname, intervals with days, months, times
grafana_message_templateReusable Go templates for notification contentname, template

An alert rule defines what to watch, what “bad” looks like, and how often to check. In other words, it tells Grafana when something is wrong. Contact points and notification policies tell it what to do about it. Together, these three resources form a complete alerting pipeline.

Alert rule group

In Terraform, alert rules live inside a rule group. A rule group is a named collection of related rules that share the same evaluation interval and folder.

ConceptWhat it is
Alert ruleThe instructions you give Grafana: what to watch, what “bad” looks like, how often to check
Rule groupA named collection of related alert rules that share the same evaluation interval and folder
Evaluation intervalHow often Grafana checks the rule’s condition
ConditionAn expression that reduces query results to a single value and compares it against a threshold

Example: a CPU usage alert

This HCL block defines a rule group with one alert rule that fires when CPU usage exceeds 80%.

The data blocks form a pipeline: the first block (A) queries the metric, the second (B) reduces the result to a single value, and the third (C) compares that value against a threshold. Blocks B and C use datasource_uid = "__expr__", which is Grafana’s built-in server-side expression engine for transforming and evaluating query results without a separate data source. The condition field on the rule points to the final data block (C), which determines when the alert fires.

Labels on the rule, like severity = "warning", control how notifications are routed. The notification policy you’ll define in the next slide matches on these labels to decide which contact point receives the alert.

hcl 
resource "grafana_rule_group" "cpu_alerts" {
  name             = "CPU alerts"
  folder_uid       = grafana_folder.alerts.uid
  interval_seconds = 60

  rule {
    name      = "High CPU usage"
    condition = "C"

    data {
      ref_id         = "A"
      datasource_uid = "prometheus"

      relative_time_range {
        from = 600
        to   = 0
      }

      model = jsonencode({
        expr = "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)"
      })
    }

    data {
      ref_id         = "B"
      datasource_uid = "__expr__"

      relative_time_range {
        from = 0
        to   = 0
      }

      model = jsonencode({
        type       = "reduce"
        expression = "A"
        reducer    = "last"
      })
    }

    data {
      ref_id         = "C"
      datasource_uid = "__expr__"

      relative_time_range {
        from = 0
        to   = 0
      }

      model = jsonencode({
        type       = "threshold"
        expression = "B"
        conditions = [{
          evaluator = {
            type   = "gt"
            params = [80]
          }
        }]
      })
    }

    labels = {
      severity = "warning"
    }
  }
}

Script

The Grafana Terraform provider gives you five resources for alerting. grafana_rule_group manages alert rule groups: the conditions, queries, and thresholds that define when an alert fires. grafana_contact_point defines where notifications go. grafana_notification_policy controls how alerts are routed to contact points. grafana_mute_timing defines time windows when notifications are suppressed. And grafana_message_template lets you customize notification content.

An alert rule defines what to watch, what “bad” looks like, and how often to check. In Terraform, you define alert rules inside a rule group. A rule group is a named collection of related alert rules that share the same evaluation interval and folder.

The condition is an expression that reduces query results to a single value and compares it against a threshold. For example, you might query CPU usage over the last five minutes, reduce it to the last value, and fire the alert when it exceeds 80 percent.

Labels on the rule control how notifications are routed. The notification policy you’ll define later matches on these labels to decide which contact point receives the alert.

Previous Next