How to provision a notification policy in Grafana Alerting — and keep it editable in the UI

Provisioning Grafana Alerting resources, such as notification policies, can help you deploy resources faster and streamline the alerting and notification process. Before getting started, it’s important to understand the different options for provisioning notification policies, how they work, and the challenges they can present.

In Grafana Alerting, notification policies use alert labels to determine how alerts are routed to different contact points or receivers. You can have multiple nested notification policies — what’s known as a tree structure — where each alert is evaluated by the default policy and, subsequently, by each nested policy. The default notification policy is also known as the root notification policy. 

To create or update a notification policy, you can either open up Grafana in your browser and click away in the UI, or provision a policy via a configuration file or API.

However, when notification policies are provisioned via an API or config file, they become non-editable in the UI. When this occurs, this message appears at the top of the provisioned notification policy:

In this post, we will walk through the steps required to provision Grafana Alerting notification policies via an API and via a configuration file. We will also cover a workaround that lets you provision a notification policy via an API and keep it editable in the UI, so you can make quick changes, as needed. (Note: This workaround is not possible when provisioning via a configuration file; it can only be accomplished via API provisioning.)

API provisioning

Again, this provisioning approach is convenient when you want to combine provisioning and editing via the UI.

Creating a new policy via an API 

To create a new policy with an API, follow the steps below: 

1. GET the notification policy tree by calling the endpoint:

/api/v1/provisioning/policies

2. You will get your root notification policy tree in JSON format. Here is a sample response:

Copy

{
	"receiver": "grafana-default",
	"routes": [
    	{
        	"receiver": "grafana-default",
        	"object_matchers": [
            	[
                	"Teams",
                	"=",
                	"True"
            	]
        	]
    	},
    	{
        	"receiver": "go-alert",
        	"object_matchers": [
            	[
                	"GoAlert",
                	"=",
                	"True"
            	]
        	]
    	}
	],
	"provenance": "file"
}

3. Make any changes you need to the original tree.
4. Before you send your PUT request (step 5 below), add the X-Disable-Provenance header. This header instructs Grafana to allow changes to be made to the policy in the UI after it has been provisioned. If used in updates, the header will only work if the policy was originally provisioned with it. This step is especially important to keep your policy editable in the UI.
5. Finally, send your PUT request with your new JSON to:

/api/v1/provisioning/policies

Updating a provisioned notification policy via an API

To update your notification policy, follow the same steps as above. On the last step, before you send your PUT request, remember to add the X-Disable-Provenance header, so the resource stays editable in the UI.

File provisioning

This approach requires you to include the whole notification policy tree in your YAML- or JSON-based configuration file. 

Creating a configuration file in YAML

When creating your config file, make sure you reference the entire policy tree by matching the receiver (line 4 in the example below). To see an example of a configuration file in YAML, check out our docs.

Copy

apiVersion: 1
policies:
  - orgId: 1
	receiver: grafana-default
	routes:
  	- receiver: grafana-default
    	object_matchers:
        	- ['Teams', '=', 'True']
  	- receiver: go-alert
    	object_matchers:
        	- ['GoAlert', '=', 'True']

If you want to add multiple policies, add them as individual entries under routes  in the policy object.

As stated earlier, the notification policy will not be editable in the UI after using this provisioning method.

Creating a configuration file in JSON

You can create a configuration file in JSON by getting your notification policy tree via this API:

Copy

GET /api/v1/provisioning/policies

To learn more about Grafana Alerting, check out our documentation.

Grafana Alerting is available for Grafana OSS, Grafana Enterprise, or Grafana Cloud. The easiest way to get started is with Grafana Cloud’s generous forever-free tier. Sign up for free now!