Documentation Grafana Cloud Developer resources Infrastructure as code Terraform Manage Asserts in Grafana Cloud using Terraform

Manage Asserts in Grafana Cloud using Terraform

Learn how to use Terraform to manage Grafana Cloud Asserts resources. This guide shows you how to create and manage alert configurations and suppressed assertions using Terraform.

Before you begin

Before you begin, you should have the following:

• A Grafana Cloud account, as shown in Get started
• Terraform installed on your machine
• Administrator permissions in your Grafana instance
• Asserts enabled in your Grafana Cloud stack

Configure the Grafana provider

This Terraform configuration sets up the Grafana provider to provide necessary authentication when managing Asserts resources.

You can reuse a similar setup to the one described in Creating and managing a Grafana Cloud stack using Terraform to set up a service account and a token.

1. Create a Service account and token in Grafana. To create a new one, refer to Service account tokens.

2. Create a file named main.tf and add the following:

   terraform Copy

   terraform {
     required_providers {
       grafana = {
         source  = "grafana/grafana"
         version = ">= 2.9.0"
       }
     }
   }
   
   provider "grafana" {
     alias = "asserts"
   
     url      = "<Stack-URL>"
     auth     = "<Service-account-token>"
     stack_id = "<Stack-ID>"
   }

3. Replace the following field values:

   • <Stack-URL> with the URL of your Grafana stack (for example, https://my-stack.grafana.net/)
   • <Service-account-token> with the service account token that you created
   • <Stack-ID> with your Grafana Cloud stack ID

Create notification alerts configurations

Notification alerts configurations in Asserts allow you to manage how alerts are processed and routed. You can specify match labels to filter alerts, add custom labels, set duration requirements, and control silencing.

Basic notification alerts configuration

Create a file named alert-configs.tf and add the following:

terraform Copy

# Basic alert configuration with silencing
resource "grafana_asserts_notification_alerts_config" "prometheus_remote_storage_failures" {
  provider = grafana.asserts

  name = "PrometheusRemoteStorageFailures"

  match_labels = {
    alertname   = "PrometheusRemoteStorageFailures"
    alertgroup  = "prometheus.alerts"
    asserts_env = "prod"
  }

  silenced = true
}

# High severity alert with specific job and context matching
resource "grafana_asserts_notification_alerts_config" "error_buildup_notify" {
  provider = grafana.asserts

  name = "ErrorBuildupNotify"

  match_labels = {
    alertname               = "ErrorBuildup"
    job                     = "acai"
    asserts_request_type    = "inbound"
    asserts_request_context = "/auth"
  }

  silenced = false
}

Notification alerts configuration with additional labels and duration

terraform Copy

# Alert with additional labels and custom duration
resource "grafana_asserts_notification_alerts_config" "payment_test_alert" {
  provider = grafana.asserts

  name = "PaymentTestAlert"

  match_labels = {
    alertname         = "PaymentTestAlert"
    additional_labels = "asserts_severity=~\"critical\""
    alertgroup        = "alex-k8s-integration-test.alerts"
  }

  alert_labels = {
    testing = "onetwothree"
  }

  duration = "5m"
  silenced = false
}

Latency and performance notification alerts

terraform Copy

# Latency alert for shipping service
resource "grafana_asserts_notification_alerts_config" "high_shipping_latency" {
  provider = grafana.asserts

  name = "high shipping latency"

  match_labels = {
    alertname            = "LatencyP99ErrorBuildup"
    job                  = "shipping"
    asserts_request_type = "inbound"
  }

  silenced = false
}

# CPU throttling alert with warning severity
resource "grafana_asserts_notification_alerts_config" "cpu_throttling_sustained" {
  provider = grafana.asserts

  name = "CPUThrottlingSustained"

  match_labels = {
    alertname         = "CPUThrottlingSustained"
    additional_labels = "asserts_severity=~\"warning\""
  }

  silenced = true
}

Infrastructure and service notification alerts

terraform Copy

# Ingress error rate alert
resource "grafana_asserts_notification_alerts_config" "ingress_error" {
  provider = grafana.asserts

  name = "ingress error"

  match_labels = {
    alertname            = "ErrorRatioBreach"
    job                  = "ingress-nginx-controller-metrics"
    asserts_request_type = "inbound"
  }

  silenced = false
}

# MySQL Galera cluster alert
resource "grafana_asserts_notification_alerts_config" "mysql_galera_not_ready" {
  provider = grafana.asserts

  name = "MySQLGaleraNotReady"

  match_labels = {
    alertname = "MySQLGaleraNotReady"
  }

  silenced = false
}

Create suppressed assertions configurations

Suppressed assertions configurations allow you to disable specific alerts or assertions based on label matching. This is useful for maintenance windows, test environments, or when you want to temporarily suppress certain types of alerts.

Create a file named suppressed-assertions.tf and add the following:

terraform Copy

# Basic suppressed alert configuration for maintenance
resource "grafana_asserts_suppressed_assertions_config" "maintenance_window" {
  provider = grafana.asserts

  name = "MaintenanceWindow"

  match_labels = {
    service     = "api-service"
    maintenance = "true"
  }
}

# Suppress specific alertname during deployment
resource "grafana_asserts_suppressed_assertions_config" "deployment_suppression" {
  provider = grafana.asserts

  name = "DeploymentSuppression"

  match_labels = {
    alertname = "HighLatency"
    job       = "web-service"
    env       = "staging"
  }
}

# Suppress alerts for specific test environment
resource "grafana_asserts_suppressed_assertions_config" "test_environment_suppression" {
  provider = grafana.asserts

  name = "TestEnvironmentSuppression"

  match_labels = {
    alertgroup  = "test.alerts"
    environment = "test"
  }
}

Resource reference

The following configurations provide more details and examples for working with the Grafana API.

grafana_asserts_notification_alerts_config

Manage Asserts notification alerts configurations through the Grafana API.

Arguments

Example

terraform Copy

resource "grafana_asserts_notification_alerts_config" "example" {
  provider = grafana.asserts

  name = "ExampleAlert"

  match_labels = {
    alertname = "HighCPUUsage"
    job       = "monitoring"
  }

  alert_labels = {
    severity = "warning"
    team     = "platform"
  }

  duration = "5m"
  silenced = false
}

grafana_asserts_suppressed_assertions_config

Manage Asserts suppressed assertions configurations through the Grafana API.

Arguments

Example

terraform Copy

resource "grafana_asserts_suppressed_assertions_config" "example" {
  provider = grafana.asserts

  name = "ExampleSuppression"

  match_labels = {
    alertname = "TestAlert"
    env       = "development"
  }
}

Apply the Terraform configuration

In a terminal, run the following commands from the directory where all of the configuration files are located.

1. Initialize a working directory containing Terraform configuration files:

   shell Copy

   terraform init

2. Preview the changes that Terraform will make:

   shell Copy

   terraform plan

3. Apply the configuration files:

   shell Copy

   terraform apply

Validation

After you apply the changes in the Terraform configurations, you should be able to verify the following:

• Notification alerts configurations are created in your Asserts instance and can be viewed in the Asserts UI
• Suppressed assertions configurations are active and suppressing the specified alerts
• The configurations are properly scoped to your Grafana Cloud stack

Best practices

When managing Asserts resources with Terraform, consider the following best practices:

Naming conventions

• Use descriptive names for your notification alerts configurations that clearly indicate their purpose
• Follow a consistent naming pattern across your organization
• Include environment or team identifiers in the names when appropriate

Label management

• Use specific and meaningful labels in match_labels to ensure precise alert filtering
• Leverage existing label conventions from your monitoring setup
• Consider using asserts_env and asserts_site labels for multi-environment setups

Silencing strategy

• Use the silenced parameter for temporary suppression rather than deleting notification alerts configurations
• Document the reason for silencing in your Terraform configuration comments
• Regularly review silenced configurations to ensure they’re still needed

Duration configuration

• Set appropriate duration values based on your alerting requirements
• Consider the nature of the monitored condition when choosing duration
• Use consistent duration formats across similar alert types

Conclusion

In this guide, you learned how to use Terraform to manage Grafana Cloud Asserts resources by creating notification alerts configurations and suppressed assertions.

To learn more about managing Grafana Cloud using Terraform, refer to Grafana provider’s documentation.

For more information about Asserts, refer to the Asserts documentation.