Set up Alerting on Grafana Labs

Upgrade Alerting

Fri, 07 Mar 2025 09:39:42 +0000

Upgrade Alerting

Warning
Legacy alerting will be removed in Grafana v11.0.0. Grafana v10.4 is the last version that offers legacy alerting and the last version of Grafana where automatic alert upgrades will be available.

Installing Grafana 11 before upgrading your legacy alerts will result in your existing alerts becoming inaccessible or lost.

For more information, refer to Legacy alerting removal: What you need to know about upgrading to Grafana Alerting.

What happens if I don’t upgrade from legacy alerting to Grafana Alerting before installing Grafana 11?

Attempting to upgrade to Grafana 11 while still having legacy alerting explicitly enabled will result in Grafana not starting and an error message informing you to either:

Downgrade to a version prior to Grafana 11 and then upgrade from legacy alerting to Grafana Alerting using one of the below upgrade methods.
Confirm that you don’t intend to upgrade your legacy alerts by disabling legacy alerting (optionally enable Grafana Alerting) in your configuration file.

Note
Confirming that you don’t intend to upgrade your legacy alerts and continuing the Grafana 11 installation may result in your legacy alerts becoming inaccessible or lost.

Grafana will start with a blank alerting slate and you will need to recreate your alerts from scratch. The below automatic upgrade methods will not be available in Grafana 11.

As of v11.0, a rolling back to v10.4 will likely restore your legacy alerts. However, this is not guaranteed to always remain the case in future versions.

Upgrade methods

Grafana provides two methods for a seamless automatic upgrade of legacy alert rules and notification channels to Grafana Alerting:

Upgrade with Preview (Recommended): Offers a safe and controlled preview environment where you can review and adjust your upgraded alerts before fully enabling Grafana Alerting.
Simple Upgrade: One-step upgrade method for specific needs where a preview environment is not essential.

Key Considerations

Feature	Upgrade with Preview	Simple Upgrade
Safety and Control	☑️ Preview environment for review and adjustment	❌ No preview, potential for unexpected issues
User Experience	☑️ Seamless transition by handling issues early	❌ Possible disruption during upgrade
Granular Control	☑️ Re-upgrade specific resources after resolving errors	❌ All or nothing upgrade, manual error correction
Stakeholder Involvement	☑️ Collaboration and review of adjusted alerts	❌ Review only available after upgrade
Provisioning Support	☑️ Configure new as-code before upgrading, simultaneous provisioning	❌ No built-in provisioning support
Simplicity	❌ May take longer to complete	☑️ Fast, one-step process
Suited for:	☑️ Complex setups, risk-averse environments, collaborative teams, heavy as-code use	☑️ Simple setups, testing environments, large fleets
Version	Grafana v10.3.0 -> v10.4.x	Grafana v9.0.0 -> v10.4.x

Note
When upgrading with either method, your legacy dashboard alerts and notification channels are copied to a new format. This is non-destructive and can be rolled back easily.

Upgrade with Preview (Recommended)

Prerequisites

Grafana v10.3.0 or later.
Grafana administrator access.
Enable alertingPreviewUpgrade feature toggle (enabled by default in v10.4.0 or later).

Suited for

Complex setups: Large deployments with intricate alert rules and notification channels.
Risk-averse environments: Situations where minimizing disruption and ensuring a smooth transition are critical.
Collaborative teams: Projects where feedback and review from stakeholders are valuable.
Heavy as-code use: Deployments with large or complex as-code configurations.

Overview

In Alerts & IRM, the Alerting section provides a preview of Grafana Alerting where you can review and modify your upgraded alerts before finalizing the upgrade.

In the Alerting (legacy) -> Alerting upgrade section, you can upgrade your existing alert rules and notification channels, and view a summary of the upgrade to Grafana Alerting.

Finalize your upgrade by restarting Grafana with the [unified_alerting] section enabled in your configuration.

Note
Alerts generated by the new alerting system are visible in the Alerting section of the navigation panel but are not active until the upgrade is finalized.

To upgrade with preview, complete the following steps.

Preview the Upgrade:
- Initiate the process: Access the upgrade functionality within Grafana by visiting the Alerting upgrade page in the Alerting (legacy) section of the navigation panel. From this page you can upgrade your existing alert rules and notification channels to the new Grafana Alerting system.
- Review the summary table: Review the detailed table outlining how your existing alert rules and notification channels were upgraded to resources in the new Grafana Alerting system.
Investigate and Resolve Errors:
- Identify errors: Carefully examine the previewed upgrade:
  - Any alert rules or notification channels that couldn’t be automatically upgraded will be highlighted with error indicators.
  - New or removed alert rules and notification channels will be highlighted with warning indicators.
- Address errors: You have two options to resolve these issues:
  - Fix legacy issues: If possible, address the problems within your legacy alerting setup and attempt to upgrade the specific resource again.
  - Create new resources: If fixing legacy issues isn’t viable, create new alert rules, notification policies, or contact points manually within the new Grafana Alerting system to replace the problematic ones.
Update As-Code Setup (Optional):
- Export upgraded resources: If you use provisioning methods to manage alert rules and notification channels, you can export the upgraded versions to generate provisioning files compatible with Grafana Alerting.
- Test new provisioning definitions: Ensure your as-code setup aligns with the new system before completing the upgrade process. Both legacy and Grafana Alerting alerts can be provisioned simultaneously to facilitate a smooth transition.
Finalize the Upgrade:
- Contact your Grafana server administrator: Once you’re confident in the state of your previewed upgrade, request to enable Grafana Alerting.
- Continued use for upgraded organizations: Organizations that have already completed the preview upgrade will seamlessly continue using their configured setup.
- Automatic upgrade for others: Organizations that haven’t initiated the upgrade with preview process will undergo the traditional automatic upgrade during this restart.
- Address issues before restart: Exercise caution, as Grafana will not start if any traditional automatic upgrades encounter errors. Ensure all potential issues are resolved before initiating this step.

Simple Upgrade

Prerequisites

Grafana v9.0.0 or later (more recent versions are recommended).

Suited for

Simple setups: Limited number of alerts and channels with minimal complexity.
Testing environments: Where a quick upgrade without a preview is sufficient.
Large fleets: Where manually reviewing each instance is not feasible.

Overview

While we recommend the Upgrade with Preview method for its enhanced safety and control, the Simple Upgrade Method exists for specific situations where a preview environment is not essential. For example, if you have a large fleet of Grafana instances and want to upgrade them all without the need to review and adjust each one individually.

Configure your Grafana instance to enable Grafana Alerting and disable legacy alerting. Then restart Grafana to automatically upgrade your existing alert rules and notification channels to the new Grafana Alerting system.

Once Grafana Alerting is enabled, you can review and adjust your upgraded alerts in the Alerting section of the navigation panel as well as export them for as-code setup.

To perform the simple upgrade, complete the following steps.

Note
Any errors encountered during the upgrade process will fail the upgrade and prevent Grafana from starting. If this occurs, you can roll back to legacy alerting.

Upgrade to Grafana Alerting:
- Enable Grafana Alerting: Modify custom configuration file.
- Restart Grafana: Restart Grafana for the configuration changes to take effect. Grafana will automatically upgrade your existing alert rules and notification channels to the new Grafana Alerting system.
Review and Adjust Upgraded Alerts:
- Review the upgraded alerts: Go to the Alerting section of the navigation panel to review the upgraded alerts.
- Export upgraded resources: If you use provisioning methods to manage alert rules and notification channels, you can export the upgraded versions to generate provisioning files compatible with Grafana Alerting.

Additional Information

Enable Grafana Alerting

Go to your custom configuration file ($WORKING_DIR/conf/custom.ini) and enter the following in your configuration:

[alerting]
enabled = false

[unified_alerting]
enabled = true

Note
If you have existing legacy alerts we advise using the Upgrade with Preview method first to ensure a smooth transition. Any organizations that have not completed the preview upgrade will automatically undergo the simple upgrade during the next restart.

Rolling back to legacy alerting

Note
For Grafana Cloud, contact customer support to enable or disable Grafana Alerting for your stack.

If you have upgraded to Grafana Alerting and want to roll back to legacy alerting, you can do so by disabling Grafana Alerting and re-enabling legacy alerting.

Go to your custom configuration file ($WORKING_DIR/conf/custom.ini) and enter the following in your configuration:

[alerting]
enabled = true

[unified_alerting]
enabled = false

This action is non-destructive. You can seamlessly switch between legacy alerting and Grafana Alerting at any time without losing any data. However, the upgrade process will only be performed once. If you have opted out of Grafana Alerting and then opt in again, Grafana will not perform the upgrade again.

If, after rolling back, you wish to delete any existing Grafana Alerting configuration and upgrade your legacy alerting configuration again from scratch, you can enable the clean_upgrade option:

[unified_alerting.upgrade]
clean_upgrade = true

Differences and limitations

There are some differences between Grafana Alerting and legacy dashboard alerts, and a number of features that are no longer supported.

Differences

Read and write access to legacy dashboard alerts are governed by the dashboard permissions (including the inherited permissions from the folder) while Grafana alerts are governed by the permissions of the folder only. During the upgrade, an alert rule might be moved to a different folder to match the permissions of the dashboard. The following rules apply:
- If the inherited dashboard permissions are different from the permissions of the folder, then the rule is moved to a new folder named after the original: <Original folder name> - <Permission Hash>.
- If the inherited dashboard permissions are the same as the permissions of the folder, then the rule is moved to the original folder.
- If the dashboard is in the General or Dashboards folder (i.e. no folder), then the rule is moved to a new General Alerting - <Permission Hash> folder.

Note
When updating your as-code provisioning setup for Grafana Alerting, newly generated folders will have a different UID from their original.

NoData and Error settings are upgraded as is to the corresponding settings in Grafana Alerting, except in two situations:
- As there is no Keep Last State option in Grafana Alerting, this option becomes either NoData or Error. If using the Simple Upgrade Method Grafana automatically creates a 1 year silence for each alert rule with this configuration. If the alert evaluation returns no data or fails (error or timeout), then it creates a special alert, which will be silenced by the silence created during the upgrade.
- Due to lack of validation, legacy alert rules imported via JSON or provisioned along with dashboards can contain arbitrary values for NoData or Error. In this situation, Grafana will use the default setting: NoData for No data, and Error for Error.
Notification channels are upgraded to an Alertmanager configuration with the appropriate routes and receivers.
Unlike legacy dashboard alerts where images in notifications are enabled per contact point, images in notifications for Grafana Alerting must be enabled in the Grafana configuration, either in the configuration file or environment variables, and are enabled for either all or no contact points.
The JSON format for webhook notifications has changed in Grafana Alerting and uses the format from Prometheus Alertmanager.
Alerting on Prometheus Both type queries is not supported in Grafana Alerting. Existing legacy alerts with Both type queries are upgraded to Grafana Alerting as alerts with Range type queries.

Limitations

Since Hipchat and Sensu notification channels are no longer supported, legacy alerts associated with these channels are not automatically upgraded to Grafana Alerting. Assign the legacy alerts to a supported notification channel so that you continue to receive notifications for those alerts.

Add an external Alertmanager

Wed, 06 Mar 2024 15:13:49 +0000

Add an external Alertmanager

Set up Grafana to use an external Alertmanager as a single Alertmanager to receive all of your alerts. This external Alertmanager can then be configured and administered from within Grafana itself.

Note
Grafana Alerting does not support sending alerts to the AWS Managed Service for Prometheus due to the lack of sigv4 support in Prometheus.

Once you have added the Alertmanager, you can use the Grafana Alerting UI to manage silences, contact points, and notification policies. A drop-down option in these pages allows you to switch between alertmanagers.

Note
Starting with Grafana 9.2, the URL configuration of external alertmanagers from the Admin tab on the Alerting page is deprecated. It will be removed in a future release.

External alertmanagers should now be configured as data sources using Grafana Configuration from the main Grafana navigation menu. This enables you to manage the contact points and notification policies of external alertmanagers from within Grafana and also encrypts HTTP basic authentication credentials that were previously visible when configuring external alertmanagers by URL.

To add an external Alertmanager, complete the following steps.

Click Connections in the left-side menu.
On the Connections page, search for Alertmanager.
Click the Create a new data source button.

If you don’t see this button, you may need to install the plugin, relaunch your Cloud instance, and then repeat steps 1 and 2.
Fill out the fields on the page, as required.

If you are provisioning your data source, set the flag handleGrafanaManagedAlerts in the jsonData field to true to send Grafana-managed alerts to this Alertmanager.

Note: Prometheus, Grafana Mimir, and Cortex implementations of Alertmanager are supported. For Prometheus, contact points and notification policies are read-only in the Grafana Alerting UI.
Click Save & test.

Provision Alerting resources

Tue, 21 May 2024 12:43:45 +0000

Provision Alerting resources

Alerting infrastructure is often complex, with many pieces of the pipeline that often live in different places. Scaling this across multiple teams and organizations is an especially challenging task. Importing and exporting (or provisioning) your alerting resources in Grafana Alerting makes this process easier by enabling you to create, manage, and maintain your alerting data in a way that best suits your organization.

You can import alert rules, contact points, notification policies, mute timings, and templates.

You cannot edit imported alerting resources in the Grafana UI in the same way as alerting resources that were not imported. You can only edit imported contact points, notification policies, templates, and mute timings in the source where they were created. For example, if you manage your alerting resources using files from disk, you cannot edit the data in Terraform or from within Grafana.

Import alerting resources

Choose from the options below to import (or provision) your Grafana Alerting resources.

Use configuration files to provision your alerting resources, such as alert rules and contact points, through files on disk.
Note
- You cannot edit provisioned resources from files in the Grafana UI.
- Provisioning with configuration files is not available in Grafana Cloud.
Use Terraform to provision alerting resources.
Use the Alerting provisioning HTTP API to manage alerting resources.

Note
The JSON output from the majority of Alerting HTTP endpoints isn’t compatible for provisioning via configuration files.

If you need the alerting resources for file provisioning, use Export Alerting endpoints to return or download them in provisioning format.

Export alerting resources

You can export both manually created and provisioned alerting resources. You can also edit and export an alert rule without applying the changes.

For detailed instructions on the various export options, refer to Export alerting resources.

View provisioned alerting resources

To view your provisioned resources in Grafana, complete the following steps.

Open your Grafana instance.
Navigate to Alerting.
Click an alerting resource folder, for example, Alert rules.

Provisioned resources are labeled Provisioned, so that it is clear that they were not created manually.

Enable alerting high availability

Tue, 21 May 2024 12:43:45 +0000

Enable alerting high availability

You can enable alerting high availability support by updating the Grafana configuration file. If you run Grafana in a Kubernetes cluster, additional steps are required. Both options are described below. Please note that the deduplication is done for the notification, but the alert will still be evaluated on every Grafana instance. This means that events in alerting state history will be duplicated by the number of Grafana instances running.

Note
If using a mix of execute_alerts=false and execute_alerts=true on the HA nodes, since the alert state is not shared amongst the Grafana instances, the instances with execute_alerts=false will not show any alert status. This is because the HA settings (ha_peers, etc), only apply to the alert notification delivery (i.e. de-duplication of alert notifications, and silences, as mentioned above).

Enable alerting high availability in Grafana using Memberlist

Before you begin

Since gossiping of notifications and silences uses both TCP and UDP port 9094, ensure that each Grafana instance is able to accept incoming connections on these ports.

To enable high availability support:

In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set [ha_peers] to the number of hosts for each Grafana instance in the cluster (using a format of host:port), for example, ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094. You must have at least one (1) Grafana instance added to the ha_peers section.
Set [ha_listen_address] to the instance IP address using a format of host:port (or the Pod’s IP in the case of using Kubernetes). By default, it is set to listen to all interfaces (0.0.0.0).
Set [ha_peer_timeout] in the [unified_alerting] section of the custom.ini to specify the time to wait for an instance to send a notification via the Alertmanager. The default value is 15s, but it may increase if Grafana servers are located in different geographic regions or if the network latency between them is high.

Enable alerting high availability in Grafana using Redis

As an alternative to Memberlist, you can use Redis for high availability. This is useful if you want to have a central database for HA and cannot support the meshing of all Grafana servers.

Make sure you have a redis server that supports pub/sub. If you use a proxy in front of your redis cluster, make sure the proxy supports pub/sub.
In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified_alerting] section.
Set ha_redis_address to the redis server address Grafana should connect to.
Optional: Set the username and password if authentication is enabled on the redis server using ha_redis_username and ha_redis_password.
Optional: Set ha_redis_prefix to something unique if you plan to share the redis server with multiple Grafana instances.

The following metrics can be used for meta monitoring, exposed by Grafana’s /metrics endpoint:

Metric	Description
alertmanager_cluster_messages_received_total	Total number of cluster messages received.
alertmanager_cluster_messages_received_size_total	Total size of cluster messages received.
alertmanager_cluster_messages_sent_total	Total number of cluster messages sent.
alertmanager_cluster_messages_sent_size_total	Total number of cluster messages received.
alertmanager_cluster_messages_publish_failures_total	Total number of messages that failed to be published.
alertmanager_cluster_members	Number indicating current number of members in cluster.
alertmanager_peer_position	Position the Alertmanager instance believes it’s in. The position determines a peer’s behavior in the cluster.
alertmanager_cluster_pings_seconds	Histogram of latencies for ping messages.
alertmanager_cluster_pings_failures_total	Total number of failed pings.

Enable alerting high availability using Kubernetes

If you are using Kubernetes, you can expose the pod IP through an environment variable via the container definition.

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Add the port 9094 to the Grafana deployment:

ports:
  - containerPort: 3000
    name: http-grafana
    protocol: TCP
  - containerPort: 9094
    name: grafana-alert
    protocol: TCP

Add the environment variables to the Grafana deployment:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Create a headless service that returns the pod IP instead of the service IP, which is what the ha_peers need:

apiVersion: v1
kind: Service
metadata:
  name: grafana-alerting
  namespace: grafana
  labels:
    app.kubernetes.io/name: grafana-alerting
    app.kubernetes.io/part-of: grafana
spec:
  type: ClusterIP
  clusterIP: 'None'
  ports:
    - port: 9094
  selector:
    app: grafana

Make sure your grafana deployment has the label matching the selector, e.g. app:grafana:
Add in the grafana.ini:

[unified_alerting]
enabled = true
ha_listen_address = "${POD_IP}:9094"
ha_peers = "grafana-alerting.grafana:9094"
ha_advertise_address = "${POD_IP}:9094"
ha_peer_timeout = 15s

Configure Alert State History

Wed, 06 Mar 2024 15:13:49 +0000

Configure Alert State History

Starting with Grafana 10, Alerting can record all alert rule state changes for your Grafana managed alert rules in a Loki instance.

This allows you to explore the behavior of your alert rules in the Grafana explore view and levels up the existing state history modal with a powerful new visualisation.

Configuring Loki

To set up alert state history, make sure to have a Loki instance Grafana can write data to. The default settings might need some tweaking as the state history modal might query up to 30 days of data.

The following change to the default configuration should work for most instances, but we recommend looking at the full Loki configuration settings and adjust according to your needs.

As this might impact the performances of an existing Loki instance, we recommend using a separate Loki instance for the alert state history.

limits_config:
  split_queries_by_interval: '24h'
  max_query_parallelism: 32

Configuring Grafana

We need some additional configuration in the Grafana configuration file to have it working with the alert state history.

The example below instructs Grafana to write alert state history to a local Loki instance:

[unified_alerting.state_history]
enabled = true
backend = "loki"
loki_remote_url = "http://localhost:3100"

[feature_toggles]
enable = alertStateHistoryLokiSecondary, alertStateHistoryLokiPrimary, alertStateHistoryLokiOnly

Adding the Loki data source

See our instructions on adding a data source.

Querying the history

If everything is set up correctly you can use the Grafana Explore view to start querying the Loki data source.

A simple litmus test to see if data is being written correctly into the Loki instance is the following query:

{ from="state-history" } | json

Performance considerations and limitations

Mon, 22 Apr 2024 09:45:38 +0000

Performance considerations and limitations

Grafana Alerting supports multi-dimensional alerting, where one alert rule can generate many alerts. For example, you can configure an alert rule to fire an alert every time the CPU of individual VMs max out. This topic discusses performance considerations resulting from multi-dimensional alerting.

Evaluating alerting rules consumes RAM and CPU to compute the output of an alerting query, and network resources to send alert notifications and write the results to the Grafana SQL database. The configuration of individual alert rules affects the resource consumption and, therefore, the maximum number of rules a given configuration can support.

The following section provides a list of alerting performance considerations.

Frequency of rule evaluation consideration. The “Evaluate Every” property of an alert rule controls the frequency of rule evaluation. We recommend using the lowest acceptable evaluation frequency to support more concurrent rules.
Cardinality of the rule’s result set. For example, suppose you are monitoring API response errors for every API path, on every VM in your fleet. This set has a cardinality of n number of paths multiplied by v number of VMs. You can reduce the cardinality of a result set - perhaps by monitoring errors-per-VM instead of for each path per VM.
Complexity of the alerting query consideration. Queries that data sources can process and respond to quickly consume fewer resources. Although this consideration is less important than the other considerations listed above, if you have reduced those as much as possible, looking at individual query performance could make a difference.

Each evaluation of an alert rule generates a set of alert instances; one for each member of the result set. The state of all the instances is written to the alert_instance table in Grafana’s SQL database. This number of write-heavy operations can cause issues when using SQLite.

Grafana Alerting exposes a metric, grafana_alerting_rule_evaluations_total that counts the number of alert rule evaluations. To get a feel for the influence of rule evaluations on your Grafana instance, you can observe the rate of evaluations and compare it with resource consumption. In a Prometheus-compatible database, you can use the query rate(grafana_alerting_rule_evaluations_total[5m]) to compute the rate over 5 minute windows of time. It’s important to remember that this isn’t the full picture of rule evaluation. For example, the load will be unevenly distributed if you have some rules that evaluate every 10 seconds, and others every 30 minutes.

These factors all affect the load on the Grafana instance, but you should also be aware of the performance impact that evaluating these rules has on your data sources. Alerting queries are often the vast majority of queries handled by monitoring databases, so the same load factors that affect the Grafana instance affect them as well.

Limited rule sources support

Grafana Alerting can retrieve alerting and recording rules stored in most available Prometheus, Loki, Mimir, and Alertmanager compatible data sources.

It does not support reading or writing alerting rules from any other data sources but the ones previously mentioned at this time.

Prometheus version support

We support the latest two minor versions of both Prometheus and Alertmanager. We cannot guarantee that older versions will work.

As an example, if the current Prometheus version is 2.31.1, we support >= 2.29.0.

The Grafana Alertmanager can only receive Grafana managed alerts

Grafana cannot be used to receive external alerts. You can only send alerts to the Grafana Alertmanager using Grafana managed alerts.

You have the option to send Grafana managed alerts to an external Alertmanager, you can find this option in the admin tab on the Alerting page.

For more information, refer to this GitHub discussion.

High load on database caused by a high number of alert instances

If you have a high number of alert instances, it can happen that the load on the database gets very high, as each state transition of an alert instance will be saved in the database.

This can be prevented by writing to the database periodically. For this the feature flag alertingSaveStatePeriodic needs to be enabled. By default it will save the states every 5 minutes to the database and on each shutdown. The periodic interval can also be configured using the state_periodic_save_interval configuration flag.

The time it takes to write to the database periodically can be monitored using the state_full_sync_duration_seconds metric that is exposed by Grafana.

If Grafana crashes or is force killed, then the database can be up to state_periodic_save_interval seconds out of date. When Grafana restarts, the UI might show incorrect state for some alerts until the alerts are re-evaluated. In some cases, alerts that were firing before the crash might fire again. If this happens, Grafana might send duplicate notifications for firing alerts.