Manage Grafana IRM in Grafana Cloud using Terraform
Learn how to use Terraform to manage Grafana IRM resources. This guide shows you how to connect an integration, configure escalation policies, and add on-call schedules using Terraform.
To illustrate the use of IRM across multiple teams, this guide features examples with two teams: Devs
and SREs
.
Additionally, it includes the necessary steps to configure Slack for IRM.
Note
Grafana IRM supports Terraform-based configuration for a limited set of resources, primarily those related to OnCall functionality. These resources use the
grafana_oncall_
naming convention in Terraform. Additional IRM components are not yet configurable via Terraform.
Before you begin
Before you begin, you should have the following:
- A Grafana Cloud account, as shown in Get started
- Terraform installed on your machine
- Administrator permissions in your Grafana instance
- (Optional) Administrator permissions in your Slack workspace, if you plan to integrate Slack with Grafana IRM
Note
All of the following Terraform configuration files should be saved in the same directory.
Connect Slack to Grafana IRM
Before including Slack settings in your Terraform setup, you must first configure the Slack integration with Grafana IRM.
To connect your Slack workspace to Grafana IRM, refer to the Slack integration for Grafana IRM documentation.
Configure the Grafana provider
This Terraform configuration sets up the Grafana provider to provide necessary authentication when managing resources for Grafana IRM.
You can reuse a similar setup to the one described in Creating and managing a Grafana Cloud stack using Terraform to set up a service account and a token.
Create a Service account and token in Grafana. To create a new one, refer to Service account tokens.
Create a file named
main.tf
and add the following:terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 3.15.3" } } } provider "grafana" { alias = "oncall" url = "<Stack-URL>" auth = "<Service-account-token>" oncall_url = "<OnCall-URL>" }
Replace the following field values:
<Stack-URL>
with the URL of your Grafana stack<Service-account-token>
with the service account token that you created<OnCall-URL>
with the API URL found on the Admin & API tab of the IRM Settings page
Note
If the service account has the right permissions, this provider setup also allows you to manage other Grafana resources.
Authentication via OnCall API tokens (deprecated)
OnCall API tokens are being deprecated. While existing tokens will continue to work, we recommend using Grafana Cloud service account tokens for all new API authentication.
To use an existing OnCall API token:
Log into your Grafana Cloud instance
Select Alerts & IRM > IRM
Click Settings, and then select Admin & API
Locate the Grafana IRM API section
View, copy or revoke existing OnCall API tokens
Create a file named
main.tf
and add the following:terraform { required_providers { grafana = { source = "grafana/grafana" version = ">= 2.9.0" } } } provider "grafana" { alias = "oncall" oncall_access_token = "<OnCall-API-Token>" oncall_url = "<OnCall-URL>" }
Replace the following field values:
<OnCall-API-Token>
with your existing OnCall API Token<OnCall-URL>
with the API URL found on the Admin & API tab of the IRM Settings page
Add on-call schedules
This Terraform configuration sets up two on-call schedules, SREs
and Devs
, using the
grafana_oncall_schedule
resource to define the schedules within Grafana IRM.
Additionally, this configuration includes Slack channels to receive notifications for the on-call schedules of each team.
To learn more about managing on-call schedules, refer to the On-call schedules documentation.
Create two new calendars in your calendar service, one for
Devs
and one forSREs
Locate and save the secret iCal URLs. For example, in a Google calendar, these URLs can be found in Settings > Settings for my calendars > Integrate calendar
Create a file named
schedule.tf
and add the following:Replace the following field values:
<Devs-channel-name>
with name of the Slack channel to notify about on-call schedules forDevs
<SREs-channel-name>
with name of the Slack channel to notify about on-call schedules forSREs
<secret-iCal-URL-for-devs-calendar>
with the secret iCal URL created in the first step forDevs
Calendar<secret-iCal-URL-for-SREs-calendar>
with the secret iCal URL created in the first step forSREs
Calendar
Add escalation chains
This Terraform configuration creates two escalation chains named SREs
and Devs
in Grafana IRM using the
grafana_oncall_escalation_chain
(Resource).
The configuration also adds the following three steps to each escalation chain using the
grafana_oncall_escalation
(Resource):
- Notify users from on-call schedule
- Wait for 5 minutes
- Notify default Slack channel
Create a file named
escalation-devs.tf
and add the following:Create a file named
escalation-sre.tf
and add the following:
Connect an integration to Grafana IRM
This Terraform configuration connects Alertmanager to Grafana IRM using the
grafana_oncall_integration
(Resource).
It also adds the Devs
escalation chain as the default route for alerts.
Create a file named
integrations.tf
and add the following:resource "grafana_oncall_integration" "AlertManager" { provider = grafana.oncall name = "AlertManager" type = "alertmanager" default_route { escalation_chain_id = grafana_oncall_escalation_chain.Devs.id } }
To configure Alertmanager, refer to Alertmanager integration for Grafana OnCall
Set up a route to configure escalation behavior for alert group notifications
This Terraform configuration sets up a route to the Alertmanager integration using the grafana_oncall_route
(Resource).
This route ensures that notifications for alerts with \"namespace\" *: *\"ops-.*\"
in the payload are escalated to the SREs
escalation chain.
Create a file named routes.tf
and add the following:
resource "grafana_oncall_route" "route_SREs" {
provider = grafana.oncall
integration_id = grafana_oncall_integration.AlertManager.id
escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
routing_regex = "\"namespace\" *: *\"ops-.*\""
position = 0
}
Apply the Terraform configuration
In a terminal, run the following commands from the directory where all of the configuration files are located.
Initialize a working directory containing Terraform configuration files.
terraform init
Preview the changes that Terraform will make.
terraform plan
Apply the configuration files.
terraform apply
Validation
After you apply the changes in the Terraform configurations, you can verify the following:
Two new Schedules named
Devs
andSREs
are created in Grafana IRM:New Escalation chain named
SREs
is created in Grafana IRM:New Escalation chain named
Devs
is created in Grafana IRM:The Alertmanager integration is added and configured with escalation policies:
Conclusion
In this guide, you learned how to use Terraform to manage Grafana IRM by connecting an integration, configuring escalation policies, and setting up on-call schedules.
To learn more about managing Grafana Cloud using Terraform, refer to Grafana provider’s documentation.