Configure alert rules and notification policies

Configuring alerts as code ensures consistent alerting policies across your organization. You can define alert rules, contact points, mute timings, and notification routing all in version-controlled Terraform files.

To configure alert rules and notification policies, complete the following steps:

1. Create a file named alerts_finance.tf for Finance team alerts:

   hcl Copy

   resource "grafana_rule_group" "finance_alerts" {
     name             = "Finance Alerts"
     folder_uid       = grafana_folder.finance.uid
     interval_seconds = 60
   
     rule {
       name      = "High Error Rate"
       condition = "A"
   
       data {
         ref_id = "A"
   
         relative_time_range {
           from = 600
           to   = 0
         }
   
         datasource_uid = grafana_data_source.testdata.uid
         model          = jsonencode({
           expr = "rate(errors[5m]) > 0.05"
         })
       }
     }
   }

2. Create a file named notif.tf for notification configuration:

   hcl Copy

   resource "grafana_contact_point" "email" {
     name = "Email Notifications"
   
     email {
       addresses = ["alerts@example.com"]
     }
   }
   
   resource "grafana_mute_timing" "monday_maintenance" {
     name = "Monday Maintenance Window"
   
     intervals {
       weekdays = ["monday"]
       times {
         start = "00:00"
         end   = "06:00"
       }
     }
   }
   
   resource "grafana_notification_policy" "root" {
     contact_point   = grafana_contact_point.email.name
     group_wait      = "30s"
     group_interval  = "5m"
     repeat_interval = "4h"
   
     mute_timings = [
       grafana_mute_timing.monday_maintenance.name
     ]
   }

3. Create a file named alerts_billing.tf with paused alerts:

   hcl Copy

   resource "grafana_rule_group" "billing_alerts" {
     name             = "Billing Alerts"
     folder_uid       = grafana_folder.it.uid
     interval_seconds = 300
   
     rule {
       name      = "High Cost Alert"
       condition = "A"
       is_paused = true
   
       // Keep the rest of the rule block from the previous step
     }
   }

4. Apply the configuration:

   sh Copy

   terraform apply

Alert rules, contact points, and notification policies are configured and active (except for paused billing alerts).

In the next milestone, you’ll learn how to verify your setup.