Set up Alerting on Grafana Labs

Add an external Alertmanager

Fri, 06 Mar 2026 07:23:54 +0000

Add an external Alertmanager

Set up Grafana to use an external Alertmanager as a single Alertmanager to receive all of your alerts. This external Alertmanager can then be configured and administered from within Grafana itself.

Once you have added the Alertmanager, you can use the Grafana Alerting UI to manage silences, contact points, and notification policies. A drop-down option in these pages allows you to switch between alertmanagers.

Note: Starting with Grafana 9.2, the URL configuration of external alertmanagers from the Admin tab on the Alerting page is deprecated. It will be removed in a future release.

External alertmanagers should now be configured as data sources using Grafana Configuration from the main Grafana navigation menu. This enables you to manage the contact points and notification policies of external alertmanagers from within Grafana and also encrypts HTTP basic authentication credentials that were previously visible when configuring external alertmanagers by URL.

To add an external Alertmanager, complete the following steps.

Click Connections in the left-side menu.
On the Connections page, search for Alertmanager.
Click the Create a new data source button.

If you don’t see this button, you may need to install the plugin, relaunch your Cloud instance, and then repeat steps 1 and 2.
Fill out the fields on the page, as required.

If you are provisioning your data source, set the flag handleGrafanaManagedAlerts in the jsonData field to true to send Grafana-managed alerts to this Alertmanager.

Note:: Prometheus, Grafana Mimir, and Cortex implementations of Alertmanager are supported. For Prometheus, contact points and notification policies are read-only in the Grafana Alerting UI.
Click Save & test.

Provision Grafana Alerting resources

Fri, 06 Mar 2026 07:23:54 +0000

Provision Grafana Alerting resources

Alerting infrastructure is often complex, with many pieces of the pipeline that often live in different places. Scaling this across multiple teams and organizations is an especially challenging task. Grafana Alerting provisioning makes this process easier by enabling you to create, manage, and maintain your alerting data in a way that best suits your organization.

There are three options to choose from:

Use file provisioning to provision your Grafana Alerting resources, such as alert rules and contact points, through files on disk.
Provision your alerting resources using the Alerting Provisioning HTTP API.

For more information on the Alerting Provisioning HTTP API, refer to Alerting provisioning API.
Provision your alerting resources using Terraform.

Note:

Currently, provisioning for Grafana Alerting supports alert rules, contact points, mute timings, and templates. Provisioned alerting resources using file provisioning or Terraform can only be edited in the source that created them and not from within Grafana or any other source. For example, if you provision your alerting resources using files from disk, you cannot edit the data in Terraform or from within Grafana.

To allow editing of provisioned resources in the Grafana UI, add the X-Disable-Provenance header to the following requests in the API:

POST /api/v1/provisioning/alert-rules
PUT /api/v1/provisioning/folder/{FolderUID}/rule-groups/{Group} (calling this endpoint will change provenance for all alert rules within the alert group)
POST /api/v1/provisioning/contact-points
POST /api/v1/provisioning/mute-timings
PUT /api/v1/provisioning/policies
PUT /api/v1/provisioning/templates/{name}

Useful Links:

Grafana provisioning

Grafana Cloud provisioning

Grafana Alerting provisioning API

Enable alerting high availability

Fri, 06 Mar 2026 07:23:54 +0000

Enable alerting high availability

You can enable alerting high availability support by updating the Grafana configuration file. If you run Grafana in a Kubernetes cluster, additional steps are required. Both options are described below. Please note that the deduplication is done for the notification, but the alert will still be evaluated on every Grafana instance. This means that events in alerting state history will be duplicated by the number of Grafana instances running.

Enable alerting high availability in Grafana

Before you begin

Since gossiping of notifications and silences uses both TCP and UDP port 9094, ensure that each Grafana instance is able to accept incoming connections on these ports.

To enable high availability support:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set [ha_peers] to the number of hosts for each Grafana instance in the cluster (using a format of host:port), for example, ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094. You must have at least one (1) Grafana instance added to the ha_peers section.
Set [ha_listen_address] to the instance IP address using a format of host:port (or the Pod’s IP in the case of using Kubernetes). By default, it is set to listen to all interfaces (0.0.0.0).

Enable alerting high availability using Kubernetes

If you are using Kubernetes, you can expose the pod IP through an environment variable via the container definition.

env:
- name: POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

Add the port 9094 to the Grafana deployment:

ports:
  - containerPort: 3000
    name: http-grafana
    protocol: TCP
  - containerPort: 9094
    name: grafana-alert
    protocol: TCP

Add the environment variables to the Grafana deployment:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Create a headless service that returns the pod IP instead of the service IP, which is what the ha_peers need:

apiVersion: v1
kind: Service
metadata:
  name: grafana-alerting
  namespace: grafana
  labels:
    app.kubernetes.io/name: grafana-alerting
    app.kubernetes.io/part-of: grafana
spec:
  type: ClusterIP
  clusterIP: 'None'
  ports:
    - port: 9094
  selector:
    app: grafana

Make sure your grafana deployment has the label matching the selector, e.g. app:grafana:
Add in the grafana.ini:

[unified_alerting]
enabled = true
ha_listen_address = "${POD_IP}:9094"
ha_peers = "grafana-alerting.grafana:9094"
ha_advertise_address = "${POD_IP}:9094"