Principles to remember

  1. Grafana resources deserve Infrastructure as Code rigor. Version control, review, and automation produce a solid observability experience.
  2. Terraform’s state management prevents drift. The state file ensures manual UI edits are corrected on the next pipeline run.
  3. The alerting pipeline has three parts. Rule groups, contact points, and notification policies work together as a unit.
  4. The pipeline is the source of truth. Code on main equals what’s deployed.

Script

Four principles to take with you.

First, Grafana resources deserve the same rigor as infrastructure. Version control, code review, automated deployment. If you do it for servers, do it for alert rules and notification policies.

Second, Terraform’s state management prevents drift and duplicates. The state file tracks what Terraform deployed, so manual UI edits get corrected on the next run.

Third, the alerting pipeline is three resources working together. Alert rule groups define when to fire, contact points define where notifications go, and notification policies define how alerts are routed.

Fourth, the pipeline is the source of truth. Once you have CI/CD in place, the code on main is always what’s deployed. Manual edits get corrected on the next run.