Set up Alerting on Grafana Labs

Upgrade Alerting

Mon, 19 Feb 2024 12:13:11 +0000

Upgrade Alerting

Grafana Alerting is enabled by default for new installations or existing installations whether or not legacy alerting is configured.

Note
When upgrading, your dashboard alerts are migrated to a new format. This migration can be rolled back easily by opting out. If you have any questions regarding this migration, please contact us.

Existing installations that do not use legacy alerting will have Grafana Alerting enabled by default unless alerting is disabled in the configuration.

Likewise, existing installations that use legacy alerting will be automatically upgraded to Grafana Alerting unless you have opted out of Grafana Alerting before migration takes place. During the upgrade, legacy alerts are migrated to the new alerts type and no alerts or alerting data are lost.

Once the upgrade has taken place, you still have the option to roll back to legacy alerting. However, we do not recommend choosing this option. If you do choose to roll back, Grafana will restore your alerts to the alerts you had at the point in time when the upgrade took place.

Note
Cloud customers, who do not want to upgrade to Grafana Alerting, should contact customer support.

If you have opted out or rolled back, you can always choose to opt in to Grafana Alerting at a later point in time.

The following table provides details on the upgrade for Cloud, Enterprise, and OSS installations and the new Grafana Alerting UI.

Grafana instance upgraded to 9.0
Cloud	Existing Cloud installations with legacy dashboard alerting will have two alerting icons in the left navigation panel - the old alerting plugin icon and the new Grafana Alerting icon. During upgrade, existing alerts from the Cloud alerting plugin are migrated to Grafana Alerting. Once migration is complete, you can access and manage the older alerts from the new alerting Grafana Alerting icon in the navigation panel. The (older) Cloud alerting plugin is uninstalled from your cloud instance. Contact customer support if you do not wish to migrate to Grafana Alerting for your Cloud stack. If you choose to use legacy alerting, use the You will see the new Grafana Alerting icon as well as the old Cloud alerting plugin in the left navigation panel.
Enterprise	Existing Enterprise instances using legacy alerting will have both the old (marked as legacy) and the new alerting icons in the navigation panel. During upgrade, existing legacy alerts are migrated to Grafana Alerting. If you wish, you can opt out of Grafana Alerting and roll back to legacy alerting. In that case, you can manage your legacy alerts from the alerting icon marked as legacy.
OSS	Existing OSS installations with legacy dashboard alerting will have two alerting icons in the left navigation panel - the old alerting icon (marked as legacy) and the new Grafana Alerting icon. During upgrade, existing legacy alerts are migrated to Grafana Alerting. If you wish, you can opt out of Grafana Alerting and roll back to legacy alerting. In that case, you can manage your legacy alerts from the alerting icon marked as legacy.

Note: Starting with v9.0, legacy alerting is deprecated and will be removed in a future release.

Opt out

You can opt out of Grafana Alerting at any time and switch to using legacy alerting. Alternatively, you can opt out of using alerting in its entirety.

Stay on legacy alerting

When upgrading to Grafana > 9.0, existing installations that use legacy alerting are automatically upgraded to Grafana Alerting unless you have opted-out of Grafana Alerting before migration takes place. During the upgrade, legacy alerts are migrated to the new alerts type and no alerts or alerting data are lost. To keep using legacy alerting and deactivate Grafana Alerting:

Go to your custom configuration file ($WORKING_DIR/conf/custom.ini).
Enter the following in your configuration:

[alerting]
enabled = true

[unified_alerting]
enabled = false

Installations that have been migrated to Grafana Alerting can roll back to legacy alerting at any time.

Note
This topic is only relevant for OSS and Enterprise customers. Contact customer support to enable or disable Grafana Alerting for your Grafana Cloud stack.

The ngalert toggle previously used to enable or disable Grafana Alerting is no longer available.

Deactivate alerting

You can deactivate both Grafana Alerting and legacy alerting in Grafana.

Go to your custom configuration file ($WORKING_DIR/conf/custom.ini).
Enter the following in your configuration:

[alerting]
enabled = false

[unified_alerting]
enabled = false

Restart Grafana for the configuration changes to take effect.

If you want to turn alerting back on, you can remove both flags to enable Grafana Alerting.

Roll back

Once the upgrade has taken place, you still have the option to roll back to legacy alerting. If you choose to roll back, Grafana will restore your alerts to the alerts you had at the point in time when the upgrade took place.

To roll back to legacy alerting, enter the following in your configuration:

[alerting]
enabled = true

[unified_alerting]
enabled = false

Note: The next time you upgrade to Grafana Alerting, Grafana will restore your Grafana Alerting alerts and configuration to those you had before rolling back.

If, after rolling back, you wish to delete any existing Grafana Alerting configuration and upgrade your legacy alerting configuration again from scratch, you can enable the clean_upgrade option:

[unified_alerting.upgrade]
clean_upgrade = true

Opt in

If you have previously disabled alerting in Grafana, or opted out of Grafana Alerting and have decided that you would now like to use Grafana Alerting, you can choose to opt in at any time.

If you have been using legacy alerting up until now your existing alerts will be migrated to the new alerts type and no alerts or alerting data are lost. Even if you choose to opt in to Grafana Alerting, you can roll back to legacy alerting at any time.

To opt in to Grafana Alerting, enter the following in your configuration:

[alerting]
enabled = false

[unified_alerting]
enabled = true

Differences and limitations

There are some differences between Grafana Alerting and legacy dashboard alerts, and a number of features that are no longer supported.

Differences

When Grafana Alerting is enabled or upgraded to Grafana 9.0 or later, existing legacy dashboard alerts migrate in a format compatible with the Grafana Alerting. In the Alerting page of your Grafana instance, you can view the migrated alerts alongside any new alerts. This topic explains how legacy dashboard alerts are migrated and some limitations of the migration.
Read and write access to legacy dashboard alerts and Grafana alerts are governed by the permissions of the folders storing them. During migration, legacy dashboard alert permissions are matched to the new rules permissions as follows:
- If there are dashboard permissions, a folder named Migrated {"dashboardUid": "UID", "panelId": 1, "alertId": 1} is created to match the permissions of the dashboard (including the inherited permissions from the folder).
- If there are no dashboard permissions and the dashboard is in a folder, then the rule is linked to this folder and inherits its permissions.
- If there are no dashboard permissions and the dashboard is in the General folder, then the rule is linked to the General Alerting folder and the rule inherits the default permissions.
NoData and Error settings are migrated as is to the corresponding settings in Grafana Alerting, except in two situations:

3.1. As there is no Keep Last State option for No Data in Grafana Alerting, this option becomes NoData. The Keep Last State option for Error is migrated to a new option Error. To match the behavior of the Keep Last State, in both cases, during the migration Grafana automatically creates a silence for each alert rule with a duration of 1 year.

3.2. Due to lack of validation, legacy alert rules imported via JSON or provisioned along with dashboards can contain arbitrary values for NoData and Error. In this situation, Grafana will use the default setting: NoData for No data, and Error for Error.
Notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any Dashboard alert go to the autogen-unlinked-channel-recv route.
Unlike legacy dashboard alerts where images in notifications are enabled per contact point, images in notifications for Grafana Alerting must be enabled in the Grafana configuration, either in the configuration file or environment variables, and are enabled for either all or no contact points.
The JSON format for webhook notifications has changed in Grafana Alerting and uses the format from Prometheus Alertmanager.
Alerting on Prometheus Both type queries is not supported in Grafana Alerting. Existing legacy alerts with Both type queries are migrated to Grafana Alerting as alerts with Range type queries.

Limitations

Since Hipchat and Sensu notification channels are no longer supported, legacy alerts associated with these channels are not automatically migrated to Grafana Alerting. Assign the legacy alerts to a supported notification channel so that you continue to receive notifications for those alerts.

Add an external Alertmanager

Tue, 24 Oct 2023 14:34:52 +0000

Add an external Alertmanager

Set up Grafana to use an external Alertmanager as a single Alertmanager to receive all of your alerts. This external Alertmanager can then be configured and administered from within Grafana itself.

Note
Grafana Alerting does not support sending alerts to the AWS Managed Service for Prometheus due to the lack of sigv4 support in Prometheus.

Once you have added the Alertmanager, you can use the Grafana Alerting UI to manage silences, contact points, and notification policies. A drop-down option in these pages allows you to switch between alertmanagers.

Note
Starting with Grafana 9.2, the URL configuration of external alertmanagers from the Admin tab on the Alerting page is deprecated. It will be removed in a future release.

External alertmanagers should now be configured as data sources using Grafana Configuration from the main Grafana navigation menu. This enables you to manage the contact points and notification policies of external alertmanagers from within Grafana and also encrypts HTTP basic authentication credentials that were previously visible when configuring external alertmanagers by URL.

To add an external Alertmanager, complete the following steps.

Click Connections in the left-side menu.
On the Connections page, search for Alertmanager.
Click the Create a new data source button.

If you don’t see this button, you may need to install the plugin, relaunch your Cloud instance, and then repeat steps 1 and 2.
Fill out the fields on the page, as required.

If you are provisioning your data source, set the flag handleGrafanaManagedAlerts in the jsonData field to true to send Grafana-managed alerts to this Alertmanager.

Note: Prometheus, Grafana Mimir, and Cortex implementations of Alertmanager are supported. For Prometheus, contact points and notification policies are read-only in the Grafana Alerting UI.
Click Save & test.

Import and export Grafana Alerting resources

Mon, 19 Feb 2024 12:13:11 +0000

Import and export Grafana Alerting resources

Alerting infrastructure is often complex, with many pieces of the pipeline that often live in different places. Scaling this across multiple teams and organizations is an especially challenging task. Importing and exporting (or provisioning) your alerting resources in Grafana Alerting makes this process easier by enabling you to create, manage, and maintain your alerting data in a way that best suits your organization.

You can import alert rules, contact points, notification policies, mute timings, and templates.

You cannot edit imported alerting resources in the Grafana UI in the same way as alerting resources that were not imported. You can only edit imported contact points, notification policies, templates, and mute timings in the source where they were created. For example, if you manage your alerting resources using files from disk, you cannot edit the data in Terraform or from within Grafana.

To modify imported alert rules, you can use the Modify export feature to edit and then export.

Choose from the options below to import your Grafana Alerting resources.

Use file provisioning to manage your Grafana Alerting resources, such as alert rules and contact points, through files on disk.

Note
File provisioning is not available in Grafana Cloud instances.
Use the Alerting Provisioning HTTP API.

For more information on the Alerting Provisioning HTTP API, refer to Alerting provisioning HTTP API.
Use Terraform.

Useful Links:

Grafana provisioning

Grafana Alerting provisioning API

Enable alerting high availability

Tue, 02 Jan 2024 23:42:12 +0000

Enable alerting high availability

You can enable alerting high availability support by updating the Grafana configuration file. If you run Grafana in a Kubernetes cluster, additional steps are required. Both options are described below. Please note that the deduplication is done for the notification, but the alert will still be evaluated on every Grafana instance. This means that events in alerting state history will be duplicated by the number of Grafana instances running.

Note
If using a mix of execute_alerts=false and execute_alerts=true on the HA nodes, since the alert state is not shared amongst the Grafana instances, the instances with execute_alerts=false will not show any alert status. This is because the HA settings (ha_peers, etc), only apply to the alert notification delivery (i.e. de-duplication of alert notifications, and silences, as mentioned above).

Enable alerting high availability in Grafana using Memberlist

Before you begin

Since gossiping of notifications and silences uses both TCP and UDP port 9094, ensure that each Grafana instance is able to accept incoming connections on these ports.

To enable high availability support:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set [ha_peers] to the number of hosts for each Grafana instance in the cluster (using a format of host:port), for example, ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094. You must have at least one (1) Grafana instance added to the ha_peers section.
Set [ha_listen_address] to the instance IP address using a format of host:port (or the Pod’s IP in the case of using Kubernetes). By default, it is set to listen to all interfaces (0.0.0.0).
Set [ha_peer_timeout] in the [unified_alerting] section of the custom.ini to specify the time to wait for an instance to send a notification via the Alertmanager. The default value is 15s, but it may increase if Grafana servers are located in different geographic regions or if the network latency between them is high.

Enable alerting high availability in Grafana using Redis

As an alternative to Memberlist, you can use Redis for high availability. This is useful if you want to have a central database for HA and cannot support the meshing of all Grafana servers.

Make sure you have a redis server that supports pub/sub. If you use a proxy in front of your redis cluster, make sure the proxy supports pub/sub.
In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set ha_redis_address to the redis server address Grafana should connect to.
[Optional] Set the username and password if authentication is enabled on the redis server using ha_redis_username and ha_redis_password.
[Optional] Set ha_redis_prefix to something unique if you plan to share the redis server with multiple Grafana instances.

The following metrics can be used for meta monitoring, exposed by Grafana’s /metrics endpoint:

Metric	Description
alertmanager_cluster_messages_received_total	Total number of cluster messages received.
alertmanager_cluster_messages_received_size_total	Total size of cluster messages received.
alertmanager_cluster_messages_sent_total	Total number of cluster messages sent.
alertmanager_cluster_messages_sent_size_total	Total number of cluster messages received.
alertmanager_cluster_messages_publish_failures_total	Total number of messages that failed to be published.
alertmanager_cluster_members	Number indicating current number of members in cluster.
alertmanager_peer_position	Position the Alertmanager instance believes it’s in. The position determines a peer’s behavior in the cluster.
alertmanager_cluster_pings_seconds	Histogram of latencies for ping messages.
alertmanager_cluster_pings_failures_total	Total number of failed pings.

Enable alerting high availability using Kubernetes

If you are using Kubernetes, you can expose the pod IP through an environment variable via the container definition.

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Add the port 9094 to the Grafana deployment:

ports:
  - containerPort: 3000
    name: http-grafana
    protocol: TCP
  - containerPort: 9094
    name: grafana-alert
    protocol: TCP

Add the environment variables to the Grafana deployment:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Create a headless service that returns the pod IP instead of the service IP, which is what the ha_peers need:

apiVersion: v1
kind: Service
metadata:
  name: grafana-alerting
  namespace: grafana
  labels:
    app.kubernetes.io/name: grafana-alerting
    app.kubernetes.io/part-of: grafana
spec:
  type: ClusterIP
  clusterIP: 'None'
  ports:
    - port: 9094
  selector:
    app: grafana

Make sure your grafana deployment has the label matching the selector, e.g. app:grafana:
Add in the grafana.ini:

[unified_alerting]
enabled = true
ha_listen_address = "${POD_IP}:9094"
ha_peers = "grafana-alerting.grafana:9094"
ha_advertise_address = "${POD_IP}:9094"
ha_peer_timeout = 15s

Configure Alert State History

Fri, 12 Jan 2024 15:34:35 +0000

Configure Alert State History

Starting with Grafana 10, Alerting can record all alert rule state changes for your Grafana managed alert rules in a Loki instance.

This allows you to explore the behavior of your alert rules in the Grafana explore view and levels up the existing state history modal with a powerful new visualisation.

Configuring Loki

To set up alert state history, make sure to have a Loki instance Grafana can write data to. The default settings might need some tweaking as the state history modal might query up to 30 days of data.

The following change to the default configuration should work for most instances, but we recommend looking at the full Loki configuration settings and adjust according to your needs.

As this might impact the performances of an existing Loki instance, we recommend using a separate Loki instance for the alert state history.

limits_config:
  split_queries_by_interval: '24h'
  max_query_parallelism: 32

Configuring Grafana

We need some additional configuration in the Grafana configuration file to have it working with the alert state history.

The example below instructs Grafana to write alert state history to a local Loki instance:

[unified_alerting.state_history]
enabled = true
backend = "loki"
loki_remote_url = "http://localhost:3100"

[feature_toggles]
enable = alertStateHistoryLokiSecondary, alertStateHistoryLokiPrimary, alertStateHistoryLokiOnly

Adding the Loki data source

See our instructions on adding a data source.

Querying the history

If everything is set up correctly you can use the Grafana Explore view to start querying the Loki data source.

A simple litmus test to see if data is being written correctly into the Loki instance is the following query:

{ from="state-history" } | json

Performance considerations and limitations

Tue, 24 Oct 2023 14:34:52 +0000

Performance considerations and limitations

Grafana Alerting supports multi-dimensional alerting, where one alert rule can generate many alerts. For example, you can configure an alert rule to fire an alert every time the CPU of individual VMs max out. This topic discusses performance considerations resulting from multi-dimensional alerting.

Evaluating alerting rules consumes RAM and CPU to compute the output of an alerting query, and network resources to send alert notifications and write the results to the Grafana SQL database. The configuration of individual alert rules affects the resource consumption and, therefore, the maximum number of rules a given configuration can support.

The following section provides a list of alerting performance considerations.

Frequency of rule evaluation consideration. The “Evaluate Every” property of an alert rule controls the frequency of rule evaluation. We recommend using the lowest acceptable evaluation frequency to support more concurrent rules.
Cardinality of the rule’s result set. For example, suppose you are monitoring API response errors for every API path, on every VM in your fleet. This set has a cardinality of n number of paths multiplied by v number of VMs. You can reduce the cardinality of a result set - perhaps by monitoring errors-per-VM instead of for each path per VM.
Complexity of the alerting query consideration. Queries that data sources can process and respond to quickly consume fewer resources. Although this consideration is less important than the other considerations listed above, if you have reduced those as much as possible, looking at individual query performance could make a difference.

Each evaluation of an alert rule generates a set of alert instances; one for each member of the result set. The state of all the instances is written to the alert_instance table in Grafana’s SQL database. This number of write-heavy operations can cause issues when using SQLite.

Grafana Alerting exposes a metric, grafana_alerting_rule_evaluations_total that counts the number of alert rule evaluations. To get a feel for the influence of rule evaluations on your Grafana instance, you can observe the rate of evaluations and compare it with resource consumption. In a Prometheus-compatible database, you can use the query rate(grafana_alerting_rule_evaluations_total[5m]) to compute the rate over 5 minute windows of time. It’s important to remember that this isn’t the full picture of rule evaluation. For example, the load will be unevenly distributed if you have some rules that evaluate every 10 seconds, and others every 30 minutes.

These factors all affect the load on the Grafana instance, but you should also be aware of the performance impact that evaluating these rules has on your data sources. Alerting queries are often the vast majority of queries handled by monitoring databases, so the same load factors that affect the Grafana instance affect them as well.

Limited rule sources support

Grafana Alerting can retrieve alerting and recording rules stored in most available Prometheus, Loki, Mimir, and Alertmanager compatible data sources.

It does not support reading or writing alerting rules from any other data sources but the ones previously mentioned at this time.

Prometheus version support

We support the latest two minor versions of both Prometheus and Alertmanager. We cannot guarantee that older versions will work.

As an example, if the current Prometheus version is 2.31.1, we support >= 2.29.0.

The Grafana Alertmanager can only receive Grafana managed alerts

Grafana cannot be used to receive external alerts. You can only send alerts to the Grafana Alertmanager using Grafana managed alerts.

You have the option to send Grafana managed alerts to an external Alertmanager, you can find this option in the admin tab on the Alerting page.

For more information, refer to this GitHub discussion.