Manage Grafana IRM in Grafana Cloud using Terraform
Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Developer resourcesbreadcrumb arrow Infrastructure as codebreadcrumb arrow Terraformbreadcrumb arrow Manage Grafana IRM in Grafana Cloud using Terraform
Grafana Cloud

Manage Grafana IRM in Grafana Cloud using Terraform

Learn how to use Terraform to manage Grafana IRM resources. This guide shows you how to connect an integration, configure escalation policies, and add on-call schedules using Terraform.

To illustrate the use of IRM across multiple teams, this guide features examples with two teams: Devs and SREs. Additionally, it includes the necessary steps to configure Slack for IRM.

Note

Grafana IRM supports Terraform-based configuration for a limited set of resources, primarily those related to OnCall functionality. These resources use the grafana_oncall_ naming convention in Terraform. Additional IRM components are not yet configurable via Terraform.

Before you begin

Before you begin, you should have the following:

  • A Grafana Cloud account, as shown in Get started
  • Terraform installed on your machine
  • Administrator permissions in your Grafana instance
  • (Optional) Administrator permissions in your Slack workspace, if you plan to integrate Slack with Grafana IRM

Note

All of the following Terraform configuration files should be saved in the same directory.

Connect Slack to Grafana IRM

Before including Slack settings in your Terraform setup, you must first configure the Slack integration with Grafana IRM.

To connect your Slack workspace to Grafana IRM, refer to the Slack integration for Grafana IRM documentation.

Configure the Grafana provider

This Terraform configuration sets up the Grafana provider to provide necessary authentication when managing resources for Grafana IRM.

You can reuse a similar setup to the one described in Creating and managing a Grafana Cloud stack using Terraform to set up a service account and a token.

  1. Create a Service account and token in Grafana. To create a new one, refer to Service account tokens.

  2. Create a file named main.tf and add the following:

    terraform
    terraform {
      required_providers {
        grafana = {
          source  = "grafana/grafana"
          version = ">= 3.15.3"
        }
      }
    }
    
    provider "grafana" {
      alias = "oncall"
    
      url  = "<Stack-URL>"
      auth = "<Service-account-token>"
      oncall_url = "<OnCall-URL>"
    }
  3. Replace the following field values:

    • <Stack-URL> with the URL of your Grafana stack
    • <Service-account-token> with the service account token that you created
    • <OnCall-URL> with the API URL found on the Admin & API tab of the IRM Settings page

Note

If the service account has the right permissions, this provider setup also allows you to manage other Grafana resources.

Authentication via OnCall API tokens (deprecated)

OnCall API tokens are being deprecated. While existing tokens will continue to work, we recommend using Grafana Cloud service account tokens for all new API authentication.

Add on-call schedules

This Terraform configuration sets up two on-call schedules, SREs and Devs, using the grafana_oncall_schedule resource to define the schedules within Grafana IRM. Additionally, this configuration includes Slack channels to receive notifications for the on-call schedules of each team.

To learn more about managing on-call schedules, refer to the On-call schedules documentation.

  1. Create two new calendars in your calendar service, one for Devs and one for SREs

  2. Locate and save the secret iCal URLs. For example, in a Google calendar, these URLs can be found in Settings > Settings for my calendars > Integrate calendar

  3. Create a file named schedule.tf and add the following:

    terraform
    # Name of the Slack channel to notify about on-call schedules for Devs
    data "grafana_oncall_slack_channel" "Devs" {
      provider = grafana.oncall
    
      name = "<Devs-channel-name>"
    }
    
    # Name of the Slack channel to notify about on-call schedules for SREs
    data "grafana_oncall_slack_channel" "SREs" {
      provider = grafana.oncall
    
      name = "<SREs-channel-name>"
    }
    
    resource "grafana_oncall_schedule" "schedule_Devs" {
      provider = grafana.oncall
    
      name             = "Devs"
      type             = "ical"
      ical_url_primary = "<secret-iCal-URL-for-devs-calendar>"
      slack {
        channel_id = data.grafana_oncall_slack_channel.Devs.slack_id
      }
    }
    
    resource "grafana_oncall_schedule" "schedule_SREs" {
      provider = grafana.oncall
    
      name             = "SREs"
      type             = "ical"
      ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>"
      slack {
        channel_id = data.grafana_oncall_slack_channel.SREs.slack_id
      }
    }
  4. Replace the following field values:

    • <Devs-channel-name> with name of the Slack channel to notify about on-call schedules for Devs
    • <SREs-channel-name> with name of the Slack channel to notify about on-call schedules for SREs
    • <secret-iCal-URL-for-devs-calendar> with the secret iCal URL created in the first step for Devs Calendar
    • <secret-iCal-URL-for-SREs-calendar> with the secret iCal URL created in the first step for SREs Calendar

Add escalation chains

This Terraform configuration creates two escalation chains named SREs and Devs in Grafana IRM using the grafana_oncall_escalation_chain (Resource). The configuration also adds the following three steps to each escalation chain using the grafana_oncall_escalation (Resource):

  • Notify users from on-call schedule
  • Wait for 5 minutes
  • Notify default Slack channel
  1. Create a file named escalation-devs.tf and add the following:

    terraform
    resource "grafana_oncall_escalation_chain" "Devs" {
      provider = grafana.oncall
    
      name = "Devs"
    }
    
    // Notify users from on-call schedule
    resource "grafana_oncall_escalation" "notify_schedule_step_Devs" {
      provider = grafana.oncall
    
      escalation_chain_id          = grafana_oncall_escalation_chain.Devs.id
      type                         = "notify_on_call_from_schedule"
      notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id
      position                     = 0
    }
    
    // Wait step for 5 Minutes
    resource "grafana_oncall_escalation" "wait_step_Devs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
      type                = "wait"
      duration            = 300
      position            = 1
    }
    
    // Notify default Slack channel step
    resource "grafana_oncall_escalation" "notify_step_Devs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
      type                = "notify_whole_channel"
      important           = true
      position            = 2
    }
  2. Create a file named escalation-sre.tf and add the following:

    terraform
    resource "grafana_oncall_escalation_chain" "SREs" {
      provider = grafana.oncall
    
      name = "SREs"
    }
    
    // Notify users from on-call schedule
    resource "grafana_oncall_escalation" "notify_schedule_step_SREs" {
      provider = grafana.oncall
    
      escalation_chain_id          = grafana_oncall_escalation_chain.SREs.id
      type                         = "notify_on_call_from_schedule"
      notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id
      position                     = 0
    }
    
    // Wait step for 5 Minutes
    resource "grafana_oncall_escalation" "wait_step_SREs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
      type                = "wait"
      duration            = 300
      position            = 1
    }
    
    // Notify default Slack channel step
    resource "grafana_oncall_escalation" "notify_step_SREs" {
      provider = grafana.oncall
    
      escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
      type                = "notify_whole_channel"
      important           = true
      position            = 2
    }

Connect an integration to Grafana IRM

This Terraform configuration connects Alertmanager to Grafana IRM using the grafana_oncall_integration (Resource). It also adds the Devs escalation chain as the default route for alerts.

  1. Create a file named integrations.tf and add the following:

    terraform
    resource "grafana_oncall_integration" "AlertManager" {
      provider = grafana.oncall
    
      name = "AlertManager"
      type = "alertmanager"
      default_route {
        escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
      }
    }
  2. To configure Alertmanager, refer to Alertmanager integration for Grafana OnCall

Set up a route to configure escalation behavior for alert group notifications

This Terraform configuration sets up a route to the Alertmanager integration using the grafana_oncall_route (Resource). This route ensures that notifications for alerts with \"namespace\" *: *\"ops-.*\" in the payload are escalated to the SREs escalation chain.

Create a file named routes.tf and add the following:

terraform
resource "grafana_oncall_route" "route_SREs" {
  provider = grafana.oncall

  integration_id      = grafana_oncall_integration.AlertManager.id
  escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
  routing_regex       = "\"namespace\" *: *\"ops-.*\""
  position            = 0
}

Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

  1. Initialize a working directory containing Terraform configuration files.

    shell
    terraform init
  2. Preview the changes that Terraform will make.

    shell
    terraform plan
  3. Apply the configuration files.

    shell
    terraform apply

Validation

After you apply the changes in the Terraform configurations, you can verify the following:

  • Two new Schedules named Devs and SREs are created in Grafana IRM:

    Devs and SREs OnCall schedules

  • New Escalation chain named SREs is created in Grafana IRM:

    SREs escalation chain

  • New Escalation chain named Devs is created in Grafana IRM:

    Devs escalation chain

  • The Alertmanager integration is added and configured with escalation policies:

    Alertmanager integration for SREs escalation

Conclusion

In this guide, you learned how to use Terraform to manage Grafana IRM by connecting an integration, configuring escalation policies, and setting up on-call schedules.

To learn more about managing Grafana Cloud using Terraform, refer to Grafana provider’s documentation.