Use configuration files to provision alerting resources
Manage your alerting resources using configuration files that can be version controlled. When Grafana starts, it provisions the resources defined in your configuration files. Provisioning can create, update, or delete existing resources in your Grafana instance.
This guide outlines the steps and references to provision alerting resources using YAML files. For a practical demo, you can clone and try this example using Grafana OSS and Docker Compose.
Note
Provisioning Grafana with configuration files is not available in Grafana Cloud.
You cannot edit provisioned resources from files in Grafana. You can only change the resource properties by changing the provisioning file and restarting Grafana or carrying out a hot reload. This prevents changes being made to the resource that would be overwritten if a file is provisioned again or a hot reload is carried out.
Provisioning using configuration files takes place during the initial set up of your Grafana system, but you can re-run it at any time using the Grafana Admin API.
Importing an existing alerting resource results in a conflict. First, when present, remove the resources you plan to import.
Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning.
Import alert rules
Create or delete alert rules using provisioning files in your Grafana instance(s).
Find the alert rule group in Grafana.
Export and download a provisioning file for your alert rules.
Copy the contents into a YAML or JSON configuration file and add it to the
provisioning/alerting
directory of the Grafana instance you want to import the alerting resources to.Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating alert rules.
Here is an example of a configuration file for deleting alert rules.
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteRules:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the rule
uid: my_id_1
Import contact points
Create or delete contact points using provisioning files in your Grafana instance(s).
Find the contact point in Grafana.
Export and download a provisioning file for your contact point.
Copy the contents into a YAML or JSON configuration file and add it to the
provisioning/alerting
directory of the Grafana instance you want to import the alerting resources to.Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating contact points.
Here is an example of a configuration file for deleting contact points.
# config file version
apiVersion: 1
# List of receivers that should be deleted
deleteContactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the receiver
uid: first_uid
Settings
Here are some examples of settings you can use for the different contact point integrations.
MQTT
type: mqtt
settings:
# <string, required>
brokerUrl: tcp://127.0.0.1:1883
# <string>
clientId: grafana
# <string, required>
topic: grafana/alerts
# <string>
messageFormat: json
# <string>
username: grafana
# <string>
password: password1
# <string>
qos: 0
# <bool>
retain: false
# <map>
tlsConfig:
# <bool>
insecureSkipVerify: false
# <string>
clientCertificate: certificate in PEM format
# <string>
clientKey: key in PEM format
# <string>
caCertificate: CA certificate in PEM format
OpsGenie
type: opsgenie
settings:
# <string, required>
apiKey: xxx
# <string, required>
apiUrl: https://api.opsgenie.com/v2/alerts
# <string>
message: |
{{ template "default.title" . }}
# <string>
description: some descriptive description
# <bool>
autoClose: false
# <bool>
overridePriority: false
# <string> options: tags, details, both
sendTagsAs: both
PagerDuty
type: pagerduty
settings:
# <string, required> the 32-character Events API key https://support.pagerduty.com/docs/api-access-keys#events-api-keys
integrationKey: XXX
# <string> options: critical, error, warning, info
severity: critical
# <string>
class: ping failure
# <string>
component: Grafana
# <string>
group: app-stack
# <string>
summary: |
{{ template "default.message" . }}
Pushover
type: pushover
settings:
# <string, required>
apiToken: XXX
# <string, required>
userKey: user1,user2
# <string>
device: device1,device2
# <string> options (high to low): 2,1,0,-1,-2
priority: '2'
# <string>
retry: '30'
# <string>
expire: '120'
# <string> the number of seconds before a message expires and is deleted automatically. Examples: 10s, 5m30s, 8h.
ttl:
# <string>
sound: siren
# <string>
okSound: magic
# <string>
message: |
{{ template "default.message" . }}
Slack
type: slack
settings:
# <string, required>
recipient: alerting-dev
# <string, required>
token: xxx
# <string>
username: grafana_bot
# <string>
icon_emoji: heart
# <string>
icon_url: https://icon_url
# <string>
mentionUsers: user_1,user_2
# <string>
mentionGroups: group_1,group_2
# <string> options: here, channel
mentionChannel: here
# <string> Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary
url: https://some_webhook_url
# <string>
endpointUrl: https://custom_url/api/chat.postMessage
# <string>
title: |
{{ template "slack.default.title" . }}
text: |
{{ template "slack.default.text" . }}
Webhook
type: webhook
settings:
# <string, required>
url: https://endpoint_url
# <string> options: POST, PUT
httpMethod: POST
# <string>
username: abc
# <string>
password: abc123
# <string>
authorization_scheme: Bearer
# <string>
authorization_credentials: abc123
# <string>
maxAlerts: '10'
# <map>
tlsConfig:
# <bool>
insecureSkipVerify: false
# <string>
clientCertificate: certificate in PEM format
# <string>
clientKey: key in PEM format
# <string>
caCertificate: CA certificate in PEM format
Import notification template groups
Create or delete notification template groups using provisioning files in your Grafana instance(s).
Find the notification template group in Grafana.
Export a template group by copying the template content and name.
Copy the contents into a YAML or JSON configuration file and add it to the
provisioning/alerting
directory of the Grafana instance you want to import the alerting resources to.Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating notification template groups.
# config file version
apiVersion: 1
# List of templates to import or update
templates:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the template group, must be unique
name: my_first_template
# <string, required> content of the template group
template: |
{{ define "my_first_template" }}
Custom notification message
{{ end }}
Here is an example of a configuration file for deleting notification template groups.
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteTemplates:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the template group, must be unique
name: my_first_template
Import notification policies
Create or reset the notification policy tree using provisioning files in your Grafana instance(s).
In Grafana, the entire notification policy tree is considered a single, large resource. Add new specific policies as sub-policies under the root policy. Since specific policies may depend on each other, you cannot provision subsets of the policy tree; the entire tree must be defined in a single place.
Warning
Since the policy tree is a single resource, provisioning it will overwrite a policy tree created through any other means.
Find the notification policy tree in Grafana.
Export and download a provisioning file for your notification policy tree.
Copy the contents into a YAML or JSON configuration file and add it to the
provisioning/alerting
directory of the Grafana instance you want to import the alerting resources to.Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating notification policies.
Here is an example of a configuration file for resetting the policy tree back to its default value:
# config file version
apiVersion: 1
# List of orgIds that should be reset to the default policy
resetPolicies:
- 1
Import mute timings
Create or delete mute timings via provisioning files using provisioning files in your Grafana instance(s).
Find the mute timing in Grafana.
Export and download a provisioning file for your mute timing.
Copy the contents into a YAML or JSON configuration file and add it to the
provisioning/alerting
directory of the Grafana instance you want to import the alerting resources to.Example configuration files can be found below.
Restart your Grafana instance (or reload the provisioned files using the Admin API).
Here is an example of a configuration file for creating mute timings.
Here is an example of a configuration file for deleting mute timings.
# config file version
apiVersion: 1
# List of mute time intervals that should be deleted
deleteMuteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
Template variable interpolation
Provisioning interpolates environment variables using the $variable
syntax.
contactPoints:
- orgId: 1
name: My Contact Email Point
receivers:
- uid: 1
type: email
settings:
addresses: $EMAIL
In this example, provisioning replaces $EMAIL
with the value of the EMAIL
environment variable or an empty string if it is not present. For more information, refer to Using environment variables in the Provision documentation.
In alerting resources, most properties support template variable interpolation, with a few exceptions:
- Alert rule annotations:
groups[].rules[].annotations
- Alert rule time range:
groups[].rules[].relativeTimeRange
- Alert rule query model:
groups[].rules[].data.model
- Mute timings name:
muteTimes[].name
- Mute timings time intervals:
muteTimes[].time_intervals[]
- Notification template group name:
templates[].name
- Notification template group content:
templates[].template
Note for properties that support interpolation, you may unexpectedly substitute template variables when not intended. To avoid this, you can escape the $variable
with $$variable
.
For example, when provisioning a subject
property in a contactPoints.receivers.settings
object that is meant to use the $labels
variable.
subject: '{{ $labels }}'
will interpolate, incorrectly defining the subject assubject: '{{ }}'
.subject: '{{ $$labels }}'
will not interpolate, correctly defining the subject assubject: '{{ $labels }}'
.
More examples
For more examples on the concept of this guide:
- Try provisioning alerting resources in Grafana OSS with YAML files through a demo project using Docker Compose or Kubernetes deployments.
- Review the distinct options about how Grafana provisions resources in the Provision Grafana documentation.
- For Helm support, review the examples provisioning alerting resources in the Grafana Helm Chart documentation.