Documentation Grafana Cloud Alerts and IRM SLO Set up Terraform Provisioning

Provision Grafana SLO Resources using Terraform

Managing SLOs can be a complex task, especially when attempting to scale this across multiple teams and organizations. Grafana SLO provisioning makes this process easier by allowing you to create, manage, and maintain your SLOs in a way that best suits your organization.

Create and manage SLO resources using Terraform

Use Terraform’s Grafana Provider to manage your SLO resources and provision them into your Grafana system. Terraform provider support for Grafana SLO makes it easy to create, manage, and maintain all your SLOs as code.

For more information on managing your SLO resources using Terraform, refer to the Grafana Provider documentation.

Complete the following tasks to create and manage your SLO resources using Terraform.

1. Create an API Key or Service Account token for provisioning.
2. Configure the Terraform provider.
3. Define your SLO resources in Terraform.
4. Run terraform apply to provision your SLO resources.

Before you begin

Ensure you have the grafana/grafana Terraform Provider 3.5.0 or higher. Ensure you have the SLO Plugin installed on your Grafana instance.

Create a Service Account Token for provisioning

It is recommended to use a Service Account Token with Editor-level permissions in order to provision your SLO resources. To create a service account token for SLO resource provisioning, refer to the documentation about creating Service Accounts.

To view all the supported authentication formats, click here.

Configure the Terraform Provider

Grafana SLO support is included as part of the Grafana Terraform provider.

The following is an example you can use to configure the Terraform provider.

terraform Copy

terraform {
    required_providers {
        grafana = {
            source = "grafana/grafana"
            version = ">= 3.5.0"
        }
    }
}

provider "grafana" {
    url = <YOUR_GRAFANA_URL>
    auth = <YOUR_GRAFANA_API_KEY>
}

Provision SLO Resources

When an SLO is created, Grafana generates recording rules, dashboards, and alerts.

Recording rules query your SLI and record it as a new metric, so that it is faster to query later.

Dashboards show the error budget, your alert status, and other metrics related to your SLO.

Alerts indicate whether you are burning through your error budget too quickly.

You can configure your SLO with or without labels, and with or without alerts. See the required schema fields here

To provision a SLO, complete the following steps.

1. Copy this sample code block into a .tf file on your local machine and customize it for the SLO you wish to create.

• Grafana SLOs can be created for Grafana Mimir data sources. To use a Grafana Mimir data source, the Ruler API must be enabled and you need either Admin or Edit permissions for the data source. Creation of an SLO requires you specify the datasourceUID. Please see the Data Source API for more information.
• team_name and service_name are special labels that are used to group SLOs in the UI. Label values must conform to k8s label syntax and character set

2. Run terraform apply to provision your SLO resources.
3. Go to the Grafana UI and check the details of your SLO.
4. If alerting rules are configured for the SLO, ensure that the Contact Points and Notification Policies are appropriately configured to ensure that alerts are sent to the appropriate members and channels.

Copy

resource "grafana_slo" "ratio" {
  name        = "HTTP Requests Kubelet Success Rate Indicator"
  description = "99.5% of  Kubelet HTTP Requests are not 5xx errors"
  query {
    ratio {
      success_metric  = "kubelet_http_requests_total{status!~\"5..\"}"
      total_metric    = "kubelet_http_requests_total"
      group_by_labels = ["job", "instance"]
    }
    type = "ratio"
  }
  objectives {
    value  = 0.995
    window = "30d"
  }
   destination_datasource { 
      uid = "grafanacloud-prom" 
  }
  label {
    key   = "team_name"
    value = "k8s-platform"
  }
  label {
    key   = "service_name"
    value = "kubelet"
  }
  alerting {
    fastburn {
      annotation {
        key   = "name"
        value = "SLO Burn Rate Very High"
      }
      annotation {
        key   = "description"
        value = "Error budget is burning too fast"
      }
    }

    slowburn {
      annotation {
        key   = "name"
        value = "SLO Burn Rate High"
      }
      annotation {
        key   = "description"
        value = "Error budget is burning too fast"
      }
    }
  }
}

Role-based access control (RBAC) permissions

The table below details the action and scope required to configure SLO RBAC permissions with Terraform.

To learn more about how to implement RBAC permissions, refer to the main topic on provisioning RBAC with Terraform.