Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Create and manage alerting resources using file provisioning
Provision your alerting resources using files from disk. When you start Grafana, the data from these files is created in your Grafana system. Grafana adds any new resources you created, updates any that you changed, and deletes old ones.
Arrange your files in a directory in a way that best suits your use case. For example, you can choose a team-based layout where every team has its own file, you can have one big file for all your teams; or you can have one file per resource type.
Details on how to set up the files and which fields are required for each object are listed below depending on which resource you are provisioning.
Note:
Provisioning takes place during the initial set up of your Grafana system, but you can re-run it at any time using the Grafana Alerting provisioning API.
Provision alert rules
Create or delete alert rules in your Grafana instance(s).
Create an alert rule in Grafana.
Use the Alerting provisioning API to extract the alert rule.
Copy the contents into a YAML or JSON configuration file in the default provisioning directory or in your configured directory.
Example configuration files can be found below.
Ensure that your files are in the right directory on the node running the Grafana server, so that they deploy alongside your Grafana instance(s).
Delete the alert rule in Grafana.
Note:
If you do not delete the alert rule, it will clash with the provisioned alert rule once uploaded.
Here is an example of a configuration file for creating alert rules.
Here is an example of a configuration file for deleting alert rules.
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteRules:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the rule
uid: my_id_1
Provision contact points
Create or delete contact points in your Grafana instance(s).
Create a YAML or JSON configuration file.
Example configuration files can be found below.
Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating contact points.
# config file version
apiVersion: 1
# List of contact points to import or update
contactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the contact point
name: cp_1
receivers:
# <string, required> unique identifier for the receiver
- uid: first_uid
# <string, required> type of the receiver
type: prometheus-alertmanager
# <object, required> settings for the specific receiver type
settings:
url: http://test:9000
Here is an example of a configuration file for deleting contact points.
# config file version
apiVersion: 1
# List of receivers that should be deleted
deleteContactPoints:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> unique identifier for the receiver
uid: first_uid
Settings
Here are some examples of settings you can use for the different contact point types.
Alertmanager
type: prometheus-alertmanager
settings:
# <string, required>
url: http://localhost:9093
# <string>
basicAuthUser: abc
# <string>
basicAuthPassword: abc123
DingDing
type: dingding
settings:
# <string, required>
url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx
# <string> options: link, actionCard
msgType: link
# <string>
message: |
{{ template "default.message" . }}
Discord
type: discord
settings:
# <string, required>
url: https://discord/webhook
# <string>
avatar_url: https://my_avatar
# <string>
use_discord_username: Grafana
# <string>
message: |
{{ template "default.message" . }}
type: email
settings:
# <string, required>
addresses: me@example.com;you@example.com
# <bool>
singleEmail: false
# <string>
message: my optional message to include
# <string>
subject: |
{{ template "default.title" . }}
Google Hangouts Chat
type: googlechat
settings:
# <string, required>
url: https://google/webhook
# <string>
message: |
{{ template "default.message" . }}
Kafka
type: kafka
settings:
# <string, required>
kafkaRestProxy: http://localhost:8082
# <string, required>
kafkaTopic: topic1
LINE
type: line
settings:
# <string, required>
token: xxx
Microsoft Teams
type: teams
settings:
# <string, required>
url: https://ms_teams_url
# <string>
title: |
{{ template "default.title" . }}
# <string>
sectiontitle: ''
# <string>
message: |
{{ template "default.message" . }}
OpsGenie
type: opsgenie
settings:
# <string, required>
apiKey: xxx
# <string, required>
apiUrl: https://api.opsgenie.com/v2/alerts
# <string>
message: |
{{ template "default.title" . }}
# <string>
description: some descriptive description
# <bool>
autoClose: false
# <bool>
overridePriority: false
# <string> options: tags, details, both
sendTagsAs: both
PagerDuty
type: pagerduty
settings:
# <string, required>
integrationKey: XXX
# <string> options: critical, error, warning, info
severity: critical
# <string>
class: ping failure
# <string>
component: Grafana
# <string>
group: app-stack
# <string>
summary: |
{{ template "default.message" . }}
Pushover
type: pushover
settings:
# <string, required>
apiToken: XXX
# <string, required>
userKey: user1,user2
# <string>
device: device1,device2
# <string> options (high to low): 2,1,0,-1,-2
priority: '2'
# <string>
retry: '30'
# <string>
expire: '120'
# <string>
sound: siren
# <string>
okSound: magic
# <string>
message: |
{{ template "default.message" . }}
Slack
type: slack
settings:
# <string, required>
recipient: alerting-dev
# <string, required>
token: xxx
# <string>
username: grafana_bot
# <string>
icon_emoji: heart
# <string>
icon_url: https://icon_url
# <string>
mentionUsers: user_1,user_2
# <string>
mentionGroups: group_1,group_2
# <string> options: here, channel
mentionChannel: here
# <string> Optionally provide a Slack incoming webhook URL for sending messages, in this case the token isn't necessary
url: https://some_webhook_url
# <string>
endpointUrl: https://custom_url/api/chat.postMessage
# <string>
title: |
{{ template "slack.default.title" . }}
text: |
{{ template "slack.default.text" . }}
Sensu Go
type: sensugo
settings:
# <string, required>
url: http://sensu-api.local:8080
# <string, required>
apikey: xxx
# <string>
entity: default
# <string>
check: default
# <string>
handler: some_handler
# <string>
namespace: default
# <string>
message: |
{{ template "default.message" . }}
Telegram
type: telegram
settings:
# <string, required>
bottoken: xxx
# <string, required>
chatid: some_chat_id
# <string>
message: |
{{ template "default.message" . }}
Threema Gateway
type: threema
settings:
# <string, required>
api_secret: xxx
# <string, required>
gateway_id: A5K94S9
# <string, required>
recipient_id: A9R4KL4S
VictorOps
type: victorops
settings:
# <string, required>
url: XXX
# <string> options: CRITICAL, WARNING
messageType: CRITICAL
Webhook
type: webhook
settings:
# <string, required>
url: https://endpoint_url
# <string> options: POST, PUT
httpMethod: POST
# <string>
username: abc
# <string>
password: abc123
# <string>
authorization_scheme: Bearer
# <string>
authorization_credentials: abc123
# <string>
maxAlerts: '10'
WeCom
type: wecom
settings:
# <string, required>
url: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxxxxx
# <string>
message: |
{{ template "default.message" . }}
# <string>
title: |
{{ template "default.title" . }}
Provision notification policies
Create or reset notification policies in your Grafana instance(s).
Create a YAML or JSON configuration file.
Example configuration files can be found below.
Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating notification policiies.
Here is an example of a configuration file for resetting notification policies.
# config file version
apiVersion: 1
# List of orgIds that should be reset to the default policy
resetPolicies:
- 1
Provision templates
Create or delete templates in your Grafana instance(s).
Create a YAML or JSON configuration file.
Example configuration files can be found below.
Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating templates.
# config file version
apiVersion: 1
# List of templates to import or update
templates:
# <int> organization ID, default = 1
- orgID: 1
# <string, required> name of the template, must be unique
name: my_first_template
# <string, required> content of the the template
template: Alerting with a custom text template
Here is an example of a configuration file for deleting templates.
# config file version
apiVersion: 1
# List of alert rule UIDs that should be deleted
deleteTemplates:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the template, must be unique
name: my_first_template
Provision mute timings
Create or delete mute timings in your Grafana instance(s).
Create a YAML or JSON configuration file.
Example configuration files can be found below.
Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
Here is an example of a configuration file for creating mute timings.
# config file version
apiVersion: 1
# List of mute time intervals to import or update
muteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
# <list> time intervals that should trigger the muting
# refer to https://prometheus.io/docs/alerting/latest/configuration/#time_interval-0
time_intervals:
- times:
- start_time: '06:00'
end_time: '23:59'
weekdays: ['monday:wednesday', 'saturday', 'sunday']
months: ['1:3', 'may:august', 'december']
years: ['2020:2022', '2030']
days_of_month: ['1:5', '-3:-1']
Here is an example of a configuration file for deleting mute timings.
# config file version
apiVersion: 1
# List of mute time intervals that should be deleted
deleteMuteTimes:
# <int> organization ID, default = 1
- orgId: 1
# <string, required> name of the mute time interval, must be unique
name: mti_1
File provisioning using Kubernetes
If you are a Kubernetes user, you can leverage file provisioning using Kubernetes configuration maps.
- Create one or more configuration maps as follows.
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-alerting
data:
provisioning.yaml: |
templates:
- name: my_first_template
template: the content for my template
- Add the file(s) to your GitOps workflow, so that they deploy alongside your Grafana instance(s).
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
name: grafana
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:latest
ports:
- name: grafana
containerPort: 3000
volumeMounts:
- mountPath: /etc/grafana/provisioning/alerting
name: grafana-alerting
readOnly: false
volumes:
- name: grafana-alerting
configMap:
defaultMode: 420
name: grafana-alerting
This eliminates the need for a persistent database to use Grafana Alerting in Kubernetes; all your provisioned resources appear after each restart or re-deployment.