Infrastructure as CodeTerraformManaging OnCall on Grafana Cloud using Terraform

Managing OnCall on Grafana Cloud using Terraform

Learn how to connect an integration to Grafana OnCall, configure escalation policies, and add your on-call schedule to Grafana OnCall using Terraform. In this guide, We will be using two hypothetical teams named Devs and SREs for showcasing how Grafana OnCall can be used by more than one team in your organization. We will also connect Slack as the ChatOps tool for Grafana OnCall in this guide.

Prerequisites

Before you begin, you should have the following available:

  • A Grafana Cloud account, as shown in Quickstarts
  • Terraform installed on your machine
  • Admin permissions in your Grafana instance as well as the Slack workspace

NOTE: All of the following Terraform configuration files should be saved in the same directory.

Connect Slack to Grafana OnCall

To install the Slack integration for Grafana OnCall, See Slack integration for Grafana OnCall

Terraform configuration for Grafana provider

This Terraform configuration configures the Grafana provider to provide necessary authentication when managing Grafana OnCall.

  1. Create an API Token in Grafana OnCall. You’ll need this key to configure Terraform.

    1. Log into your Grafana Cloud instance.
    2. Open Grafana OnCall and select the Settings(Gear icon) tab.
    3. Under API Tokens, Click +Create.
    4. Enter a name for your API Token and click Create.
    5. Click Copy Token to save it for later use.
  2. Create a file named main.tf and add the following code block:

    terraform {
      required_providers {
        grafana = {
          source  = "grafana/grafana"
          version = "1.24.0"
        }
      }
    }
    
    provider "grafana" {
      alias = "oncall"
    
      oncall_access_token = "<OnCall-API-Token>"
    }
    
  3. Replace the following field values:

    • <Oncall-API-Token> with API Token created in the first step.

Add on-call schedule

This Terraform configuration creates two on-call schedules named SREs and Devs in Grafana OnCall using grafana_oncall_schedule (Resource). It also configures Slack channels to be notified about on-call schedules for both Devs and SREs. See more about managing on-call schedules

  1. Create two new calendars in your calendar service, one for Devs and one for SREs.

  2. Locate and save the secret iCal URLs. For example, in a Google calendar, these URLs can be found in Settings > Settings for my calendars > Integrate calendar.

  3. Create a file named schedule.tf and add the following:

    # Name of the Slack channel to notify about on-call schedules for Devs
    data "grafana_oncall_slack_channel" "Devs" {
      provider = grafana.oncall
    
      name = "<Devs-channel-name>"
    }
    
    # Name of the Slack channel to notify about on-call schedules for SREs
    data "grafana_oncall_slack_channel" "SREs" {
      provider = grafana.oncall
    
      name = "<SREs-channel-name>"
    }
    
    resource "grafana_oncall_schedule" "schedule_Devs" {
      provider = grafana.oncall
    
      name             = "Devs"
      type             = "ical"
      ical_url_primary = "<secret-iCal-URL-for-devs-calendar>"
      slack {
        channel_id = data.grafana_oncall_slack_channel.Devs.slack_id
      }
    }
    
    resource "grafana_oncall_schedule" "schedule_SREs" {
      provider = grafana.oncall
    
      name             = "SREs"
      type             = "ical"
      ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>"
      slack {
        channel_id = data.grafana_oncall_slack_channel.SREs.slack_id
      }
    }
    
  4. Replace the following field values:

    • <Devs-channel-name> with name of the Slack channel to notify about on-call schedules for Devs
    • <SREs-channel-name> with name of the Slack channel to notify about on-call schedules for SREs
    • <secret-iCal-URL-for-devs-calendar> with the secret iCal URL created in the first step for Devs Calendar
    • <secret-iCal-URL-for-SREs-calendar> with the secret iCal URL created in the first step for SREs Calendar

Add Escalation chains

This Terraform configuration creates two escalation chains named SREs and Devs in Grafana OnCall using grafana_oncall_escalation_chain (Resource). The configuration also adds the following three steps to the two escalation chains using grafana_oncall_escalation (Resource):

  • Notify users from on-call schedule
  • Wait for 5 minutes
  • Notify default Slack channel
  1. Create a file named escalation-devs.tf and add the following:

    resource "grafana_oncall_escalation_chain" "Devs" {
      provider = grafana.oncall
    
      name = "Devs"
    }
    
    // Notify users from on-call schedule
    resource "grafana_oncall_escalation" "notify_schedule_step_Devs" {
      provider = grafana.oncall
    
      escalation_chain_id          = grafana_oncall_escalation_chain.Devs.id
      type                         = "notify_on_call_from_schedule"
      notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id
      position                     = 0
    }
    
    // Wait step for 5 Minutes
    resource "grafana_oncall_escalation" "wait_step_Devs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
      type                = "wait"
      duration            = 300
      position            = 1
    }
    
    // Notify default Slack channel step
    resource "grafana_oncall_escalation" "notify_step_Devs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
      type                = "notify_whole_channel"
      important           = true
      position            = 2
    }
    
  2. Create a file named escalation-sre.tf and add the following:

    resource "grafana_oncall_escalation_chain" "SREs" {
      provider = grafana.oncall
    
      name = "SREs"
    }
    
    // Notify users from on-call schedule
    resource "grafana_oncall_escalation" "notify_schedule_step_SREs" {
      provider = grafana.oncall
    
      escalation_chain_id          = grafana_oncall_escalation_chain.SREs.id
      type                         = "notify_on_call_from_schedule"
      notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id
      position                     = 0
    }
    
    // Wait step for 5 Minutes
    resource "grafana_oncall_escalation" "wait_step_SREs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
      type                = "wait"
      duration            = 300
      position            = 1
    }
    
    // Notify default Slack channel step
    resource "grafana_oncall_escalation" "notify_step_SREs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
      type                = "notify_whole_channel"
      important           = true
      position            = 2
    }
    

Connect an integration to Grafana OnCall

This Terraform configuration connects AlertManager to Grafana OnCall using grafana_oncall_integration (Resource). It also adds the Devs escalation chain as the default route for alerts.

  1. Create a file named integrations.tf and add the following:

    resource "grafana_oncall_integration" "AlertManager" {
      provider = grafana.oncall
    
      name = "AlertManager"
      type = "alertmanager"
      default_route {
        escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
      }
    }
    
  2. To configure AlertManager see Connect AlertManager to Grafana OnCall

Set up a route to configure escalation behavior for alert group notifications

This Terraform configuration sets up a route to the AlertManager integration using grafana_oncall_route (Resource). This route ensures that notifications for alerts with \"namespace\" *: *\"ops-.*\" in the payload will be escalated to the SREs escalation chain.

Create a file named routes.tf and add the following:

resource "grafana_oncall_route" "route_SREs" {
  provider = grafana.oncall

  integration_id      = grafana_oncall_integration.AlertManager.id
  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  routing_regex       = "\"namespace\" *: *\"ops-.*\""
  position            = 0
}

Validation

Once you apply the changes in the Terraform configurations, you can verify the following:

  • Two new Schedules named Devs and SREs are created in Grafana OnCall.

    Cloud Portal

  • New Escalation chain named SREs is created in Grafana OnCall

    API Key

  • New Escalation chain named Devs is created in Grafana OnCall

    API Key

  • The AlertManager integration is added and configured with escalation policies

    InfluxDB datasource

Conclusion

In this guide, you connected an integration to Grafana OnCall, configured escalation policies, and added your on-call schedule to Grafana OnCall using Terraform. We also connected Slack as the ChatOps tool for Grafana OnCall.

To learn more about managing Grafana Cloud using Terraform, see Grafana provider’s documentation.