Slide 5 of 6

Alerting pipeline in Terraform

How the alerting pipeline works

The notification policy tree starts with a default route. All alerts enter at the top and flow down through nested policies until they match. The first matching policy determines which contact point receives the notification.

Labels on the alert rule, like severity = "warning", are what the notification policy matches against. This is why consistent labeling matters: it’s the contract between your alert rules and your notification routing.

Example: a Slack contact point

A contact point defines a notification destination. This example sends alerts to a Slack channel.

hcl 
resource "grafana_contact_point" "slack_alerts" {
  name = "Slack alerts"

  slack {
    url     = var.slack_webhook_url
    title   = "{{ .CommonLabels.alertname }}"
    text    = "{{ .CommonAnnotations.description }}"
  }
}

Example: a notification policy

A notification policy routes alerts to contact points based on label matching. This example routes critical alerts to PagerDuty and everything else to Slack.

hcl 
resource "grafana_notification_policy" "main" {
  contact_point = grafana_contact_point.slack_alerts.name
  group_by      = ["alertname"]

  policy {
    matcher {
      label = "severity"
      match = "="
      value = "critical"
    }
    contact_point = grafana_contact_point.pagerduty.name
  }
}

Script

An alert rule tells Grafana when something is wrong. But it doesn’t say what to do about it. That’s the job of contact points and notification policies.

A contact point defines where notifications go: email, Slack, PagerDuty, or any other supported integration. In Terraform, you define a contact point with the grafana_contact_point resource.

A notification policy controls how alerts are routed to contact points. It matches on alert labels. For example, a policy might route all alerts with severity equals critical to PagerDuty, and everything else to a Slack channel. In Terraform, you define this with the grafana_notification_policy resource.

Together, these three resources, the rule group, the contact point, and the notification policy, form a complete alerting pipeline. You define all three in HCL, and Terraform deploys them as a unit.

Previous Next