Documentation Grafana documentation Alerting Set up Upgrade Alerting

Upgrade Alerting

Grafana provides two methods for a seamless automatic upgrade of legacy alert rules and notification channels to Grafana Alerting:

1. Upgrade with Preview (Recommended): Offers a safe and controlled preview environment where you can review and adjust your upgraded alerts before fully enabling Grafana Alerting.
2. Simple Upgrade: One-step upgrade method for specific needs where a preview environment is not essential.

Key Considerations

Upgrade with Preview (Recommended)

Prerequisites

• Grafana v10.3.0 or later.
• Grafana administrator access.
• Enable alertingPreviewUpgrade feature toggle (enabled by default in v10.4.0 or later).

Suited for

• Complex setups: Large deployments with intricate alert rules and notification channels.
• Risk-averse environments: Situations where minimizing disruption and ensuring a smooth transition are critical.
• Collaborative teams: Projects where feedback and review from stakeholders are valuable.
• Heavy as-code use: Deployments with large or complex as-code configurations.

Overview

In Alerts & IRM, the Alerting section provides a preview of Grafana Alerting where you can review and modify your upgraded alerts before finalizing the upgrade.

In the Alerting (legacy) -> Alerting upgrade section, you can upgrade your existing alert rules and notification channels, and view a summary of the upgrade to Grafana Alerting.

Finalize your upgrade by restarting Grafana with the [unified_alerting] section enabled in your configuration.

To upgrade with preview, complete the following steps.

1. Preview the Upgrade:
   • Initiate the process: Access the upgrade functionality within Grafana by visiting the Alerting upgrade page in the Alerting (legacy) section of the navigation panel. From this page you can upgrade your existing alert rules and notification channels to the new Grafana Alerting system.
   • Review the summary table: Review the detailed table outlining how your existing alert rules and notification channels were upgraded to resources in the new Grafana Alerting system.
2. Investigate and Resolve Errors:
   • Identify errors: Carefully examine the previewed upgrade:
     • Any alert rules or notification channels that couldn’t be automatically upgraded will be highlighted with error indicators.
     • New or removed alert rules and notification channels will be highlighted with warning indicators.
   • Address errors: You have two options to resolve these issues:
     • Fix legacy issues: If possible, address the problems within your legacy alerting setup and attempt to upgrade the specific resource again.
     • Create new resources: If fixing legacy issues isn’t viable, create new alert rules, notification policies, or contact points manually within the new Grafana Alerting system to replace the problematic ones.
3. Update As-Code Setup (Optional):
   • Export upgraded resources: If you use provisioning methods to manage alert rules and notification channels, you can export the upgraded versions to generate provisioning files compatible with Grafana Alerting.
   • Test new provisioning definitions: Ensure your as-code setup aligns with the new system before completing the upgrade process. Both legacy and Grafana Alerting alerts can be provisioned simultaneously to facilitate a smooth transition.
4. Finalize the Upgrade:
   • Contact your Grafana server administrator: Once you’re confident in the state of your previewed upgrade, request to enable Grafana Alerting.
   • Continued use for upgraded organizations: Organizations that have already completed the preview upgrade will seamlessly continue using their configured setup.
   • Automatic upgrade for others: Organizations that haven’t initiated the upgrade with preview process will undergo the traditional automatic upgrade during this restart.
   • Address issues before restart: Exercise caution, as Grafana will not start if any traditional automatic upgrades encounter errors. Ensure all potential issues are resolved before initiating this step.

Simple Upgrade

Prerequisites

• Grafana v9.0.0 or later (more recent versions are recommended).

Suited for

• Simple setups: Limited number of alerts and channels with minimal complexity.
• Testing environments: Where a quick upgrade without a preview is sufficient.
• Large fleets: Where manually reviewing each instance is not feasible.

Overview

While we recommend the Upgrade with Preview method for its enhanced safety and control, the Simple Upgrade Method exists for specific situations where a preview environment is not essential. For example, if you have a large fleet of Grafana instances and want to upgrade them all without the need to review and adjust each one individually.

Configure your Grafana instance to enable Grafana Alerting and disable legacy alerting. Then restart Grafana to automatically upgrade your existing alert rules and notification channels to the new Grafana Alerting system.

Once Grafana Alerting is enabled, you can review and adjust your upgraded alerts in the Alerting section of the navigation panel as well as export them for as-code setup.

To perform the simple upgrade, complete the following steps.

1. Upgrade to Grafana Alerting:
   • Enable Grafana Alerting: Modify custom configuration file.
   • Restart Grafana: Restart Grafana for the configuration changes to take effect. Grafana will automatically upgrade your existing alert rules and notification channels to the new Grafana Alerting system.
2. Review and Adjust Upgraded Alerts:
   • Review the upgraded alerts: Go to the Alerting section of the navigation panel to review the upgraded alerts.
   • Export upgraded resources: If you use provisioning methods to manage alert rules and notification channels, you can export the upgraded versions to generate provisioning files compatible with Grafana Alerting.

Additional Information

Enable Grafana Alerting

Go to your custom configuration file ($WORKING_DIR/conf/custom.ini) and enter the following in your configuration:

toml Copy

[alerting]
enabled = false

[unified_alerting]
enabled = true

Rolling back to legacy alerting

If you have upgraded to Grafana Alerting and want to roll back to legacy alerting, you can do so by disabling Grafana Alerting and re-enabling legacy alerting.

Go to your custom configuration file ($WORKING_DIR/conf/custom.ini) and enter the following in your configuration:

toml Copy

[alerting]
enabled = true

[unified_alerting]
enabled = false

This action is non-destructive. You can seamlessly switch between legacy alerting and Grafana Alerting at any time without losing any data. However, the upgrade process will only be performed once. If you have opted out of Grafana Alerting and then opt in again, Grafana will not perform the upgrade again.

If, after rolling back, you wish to delete any existing Grafana Alerting configuration and upgrade your legacy alerting configuration again from scratch, you can enable the clean_upgrade option:

toml Copy

[unified_alerting.upgrade]
clean_upgrade = true

Differences and limitations

There are some differences between Grafana Alerting and legacy dashboard alerts, and a number of features that are no longer supported.

Differences

1. Read and write access to legacy dashboard alerts are governed by the dashboard permissions (including the inherited permissions from the folder) while Grafana alerts are governed by the permissions of the folder only. During the upgrade, an alert rule might be moved to a different folder to match the permissions of the dashboard. The following rules apply:

   • If the inherited dashboard permissions are different from the permissions of the folder, then the rule is moved to a new folder named after the original: <Original folder name> - <Permission Hash>.
   • If the inherited dashboard permissions are the same as the permissions of the folder, then the rule is moved to the original folder.
   • If the dashboard is in the General or Dashboards folder (i.e. no folder), then the rule is moved to a new General Alerting - <Permission Hash> folder.

2. NoData and Error settings are upgraded as is to the corresponding settings in Grafana Alerting, except in two situations:

   • As there is no Keep Last State option in Grafana Alerting, this option becomes either NoData or Error. If using the Simple Upgrade Method Grafana automatically creates a 1 year silence for each alert rule with this configuration. If the alert evaluation returns no data or fails (error or timeout), then it creates a special alert, which will be silenced by the silence created during the upgrade.
   • Due to lack of validation, legacy alert rules imported via JSON or provisioned along with dashboards can contain arbitrary values for NoData or Error. In this situation, Grafana will use the default setting: NoData for No data, and Error for Error.

3. Notification channels are upgraded to an Alertmanager configuration with the appropriate routes and receivers.

4. Unlike legacy dashboard alerts where images in notifications are enabled per contact point, images in notifications for Grafana Alerting must be enabled in the Grafana configuration, either in the configuration file or environment variables, and are enabled for either all or no contact points.

5. The JSON format for webhook notifications has changed in Grafana Alerting and uses the format from Prometheus Alertmanager.

6. Alerting on Prometheus Both type queries is not supported in Grafana Alerting. Existing legacy alerts with Both type queries are upgraded to Grafana Alerting as alerts with Range type queries.

Limitations

1. Since Hipchat and Sensu notification channels are no longer supported, legacy alerts associated with these channels are not automatically upgraded to Grafana Alerting. Assign the legacy alerts to a supported notification channel so that you continue to receive notifications for those alerts.