Documentation Grafana Cloud Alerts and IRM IRM Configure Integrations IRM integration for Slack

Grafana IRM integration for Slack

The Grafana IRM integration for Slack brings your incident response and on-call workflows directly into your Slack workspace, helping your team focus on alert resolution with less friction.

Our integration for Slack is available on all plans, including our free tier. More information on pricing can be found on our pricing page.

Key features and benefits

Integrating your Slack workspace allows users and teams to be notified of alerts directly in Slack with automated alert escalation steps and user notification preferences. Users can take a number of alert actions directly from Slack, including acknowledging and resolving alert groups, adding resolution notes, and more.

Refer to the following sections to learn more:

• Configure Escalation Chains with Slack notifications
• Configure user notification rules with Slack mentions
• Configure on-call shift notifications

When it comes to incidents, users can declare, collaborate on, and manage incident-worthy events without leaving Slack. They can also automatically create incident-specific channels, track the timeline of events, interact with incidents via Slash commands, and more.

Refer to the following sections to learn more:

• Create incident Slack channels
• Configure incident announcements
• Interact with incidents from Slack via Slash commands

Before you begin

To install the Grafana IRM app for Slack, you must:

• Have an Admin role in Grafana IRM
• Be a Slack workspace Admin or Owner
• Allow Grafana IRM to access your Slack workspace

After the IRM app has been added to your Slack workspace, Admins and Editors can configure escalation chains and other notifications to route to designated Slack channels.

In order to use the slash command /grafana, you must be an IRM user with a linked Slack account and necessary permissions.

To learn more, refer to Connect your Slack user to Grafana IRM.

Install the Grafana IRM integration for Slack

The Grafana IRM integration for Slack is installed and managed from the unified IRM app in Grafana Cloud. Installing the integration enables Slack notifications and incident collaboration features across all IRM workflows.

Follow these steps to install the integration:

1. In Grafana Cloud, open the main menu and navigate to Alerts & IRM > IRM > Integrations.
2. Click the Apps tab and select the Slack.
3. Click Install integration.
4. After being redirected to the Slack connection page, verify the Slack workspace is correct or Add another workspace.
5. If needed, provide your Slack workspace URL and sign in with your Slack credentials.
6. Follow the Slack prompts to review permissions and specify a default channel.
7. Click Allow to give Grafana IRM permission to access your Slack workspace.

If you already have the legacy OnCall or Incident integrations for Slack installed, use the Migration guide to migrate your existing Slack configurations.

For more information about the required permissions, refer to the Permissions section.

Connect your Slack profile to Grafana IRM

For users to gain full access to Grafana IRM functionality in Slack, follow these steps to map your Grafana IRM user account to your Slack user account:

1. In Grafana Cloud, click your avatar and go to your Profile.
2. Select the IRM tab.
3. In the Slack connection section, click Open Slack connection page.
4. Authorize your Slack account and follow the prompts to verify and link your Slack account.

After linking your Slack account, you can start using Grafana IRM in Slack in accordance with the role and permissions assigned to you in Grafana. The actions you can take in Slack depend on your permissions in Grafana IRM.

For more information on roles and permissions, refer to the Grafana Cloud user roles and permissions documentation.

Disconnect Slack profile from IRM

If you need to unlink your Slack profile from a Grafana Cloud stack, you can do so from the same tab in your Grafana user profile. Grafana Cloud Admins can also manage linked Slack profile on behalf of users in their organization. For more information, refer to Manage linked Slack profiles.

Configure channels and notifications

Grafana IRM provides flexible configuration options to tailor Slack notifications to your team’s needs. This section outlines how to set up escalation chains, user notifications, on-call notifications, and incident channel settings within Slack.

Configure Escalation Chains with Slack notifications

After setting up your integration, you can configure escalation chains to send notifications via Slack for firing alerts in Grafana IRM.

There are two Slack notification methods that can be integrated into escalation chains:

• Notify all members of a Slack channel
• Notify a specific Slack user group

To configure these in your escalation chains:

1. In Grafana Cloud, navigate to Alerts & IRM > IRM > Escalation chains.
2. Select an existing escalation chain or create a new one by clicking + Add escalation chain.
3. Use the dropdown under Add escalation step to choose and configure your Slack notification preferences.

Configure user notification rules with Slack mentions

Before configuring Slack notifications, ensure you have connected your Slack account as described in Connect notification channels.

To configure Slack mentions for your notification rules:

1. Go to your Grafana Cloud Profile.
2. Select the IRM tab.
3. Click Notification rules.
4. Click + Add notification rule.
5. Choose Slack mentions from the notification channel dropdown.
6. Configure the remaining notification rule settings as needed.

If you don’t see Slack as an available notification channel in your user profile, it may be disabled by your organization administrator. For more information about notification channel visibility and management, refer to the Notification channel documentation.

Configure on-call shift notifications

The IRM integration for Slack also supports automated notifications for on-call shifts, helping teams stay informed of their duties. It can also automatically update Slack user group handles to reflect current on-call assignments, ensuring that @mentions and notifications always reach the right people.

Admins can set up these notifications either in Grafana IRM or directly in Slack.

Configure on-call shift notifications in Grafana IRM

1. In Grafana Cloud, navigate to Alerts & IRM > IRM > Schedules.
2. In the row for the schedule you want to configure, click the gear icon to open Schedule settings.
3. Expand the Notifications settings section.
4. Configure the notification behavior for future shifts according to your team’s preferences.
5. Click Update Schedule to save your changes.

Configure on-call shift notifications directly in Slack

1. When an on-call shift notification is sent to a person or channel, click the gear icon to open Notifications preferences.
2. Configure the notification behavior for future shifts according to your team’s preferences.
3. Click Close.

Configure Incident Slack channels

Configure the following incident-specific Slack channel settings to suit your organization’s process and preferences.

Incident channel creation

Customize when incident Slack channels are created to align with your organization’s workflows.

To configure incident channel creation:

1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack.
2. Specify the following settings:
   • Create Slack channel: Determine when to automatically create a Slack channel for an incident.
   • Post updates to channel: Choose whether to automatically post incident updates to the created Slack channel.
   • Invite users: Specify if users with an active role should be automatically invited to the incident channel.
3. For private incidents, specify a default private Slack channel to post updates about any private incidents that have been created.

Automatic channel archive

To maintain good Slack workspace hygiene, you can automatically archive resolved incident channels after a configurable number of days.

Consider a timeframe of 30-60 days to allow time for PIR, follow-ups, and stakeholder visibility.

To configure automatic channel archive:

1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack.
2. Set the desired number of days after incident resolution for channels to automatically archive.
3. Click Save settings.

Channel name template

You can customize incident Slack channel names by configuring the channel name template. This allows you to standardize channel naming conventions across your organization.

By default, the template is: {{ prefix }}-{{ date }}-{{ title }}

This creates channel names like incident-2024-01-12-database-outage.

Available template variables:

• {{ prefix }} - The incident prefix, configurable in incident settings
• {{ title }} - The incident title, automatically slugified for Slack compatibility
• {{ date }} - The incident creation date in YYYY-MM-DD format
• {{ id }} - The unique incident ID

To configure the channel name template:

1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack.
2. In the Channel name template field, enter your desired template using the available variables.
3. Click Save settings.

Updating the channel name template only affects incidents created after the change. Existing incident channels are not renamed retroactively.

Example templates:

• {{ prefix }}-{{ date }}-{{ title }} produces incident-2024-01-15-database-outage
• {{ title }}-{{ id }} produces database-outage-inc-123456
• ops-{{ title }} produces ops-database-outage

Customize Slack channel prefixes

You can define custom channel prefixes to organize and identify incident channels more effectively.

To customize Slack channel prefixes:

1. In Grafana Cloud, navigate to Alerts & IRM > IRM > Settings.
2. Select the Incidents tab and find the Prefixes section.
3. Click + Add Prefix to create a new prefix.
4. Enter a Name and Description for the prefix.
5. Edit or delete existing prefixes as needed.
6. When declaring a new incident, select the desired prefix from the dropdown menu.

For more information, refer to the Customize incident channel prefixes documentation.

Configure Incident announcements

Incident announcements help keep your team aware and informed during critical events. Configure these announcements to ensure that stakeholders are kept up-to-date in Slack during an incident.

To configure incident announcements:

1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack.
2. Select the Slack channel where you want to send notifications. You can either choose from the dropdown menu or manually add your Channel ID. The Channel ID can be found in the About tab of the Slack channel.
3. Define the incident-specific fields:
   • Incident type: Choose whether the incident is internal or private.
   • Include incidents: Specify which types of incidents to announce—options include all incidents, drills, or non-drills.
4. Apply filters to your incident announcements to tailor the notifications to specific channels:
   • Incident filter: For example, you might filter by label, such as label: 'squad:datasources'. By default, all conditions are applied using AND logic. To use OR logic, you must use parentheses. For example: or(label:"squad:team-a" label:"team-b") status:active filters for (team-a OR team-b) AND status:active. Refer to the Incident query syntax documentation for full syntax availability.

Use Slack reactions

You can use emoji reactions to add context or updates to an incident without leaving Slack.

The following reactions are supported:

Additional details:

• Removing the 🤖 (:robot:) reaction deletes any attached files from Grafana Cloud storage.
• Deleting an item from the timeline within the IRM web app also removes the associated file from Grafana Cloud storage.
• Files in Grafana IRM are limited to 100MB. Files too large to persist are still accessible via a link to the source file in Slack.
• You cannot add new status updates to incidents that have been resolved.

Available commands for Slack

The /grafana commands for Slack allow users and teams to respond to alert groups and collaborate on incidents directly from Slack.

These commands can be used in any Slack channel.

Incident-related commands

Incidents management

Manage incidents, can be used in any channel.

In-Channel Incident Management

Manage the current incident, can be used in the incident channel.

Tasks

Manage tasks for an incident, can be used in the incident channel.

Notes

Manage notes for an incident, can be used in the incident channel.

Permissions scope and purpose

The Grafana IRM app for Slack only requests permissions that are essential to its proper function and integration with Slack. Refer to the Slack documentation for more information on permission scopes.

By granting access to the app, you are authorizing Grafana IRM the following permissions in your Slack workspace:

Workspace and user access

Public channel access

Private channel access

Message and conversation access

Usage limits

Grafana IRM can be rate limited by Slack when posting a high volume of messages. This typically happens during periods of frequent alert activity. For example, if updates to alert groups are rate-limited, you’ll see a message similar to the following:

Copy

Update Alert Groups in Slack, for integration xxxx, is temporarily rate-limited (due to a Slack rate-limit). Meanwhile, you can still find new Alert Groups in the Alert Groups web page.

Slack enforces rate limits per channel and per organization. When IRM hits a rate limit:

• The affected integration is blocked from posting or updating alert groups for 5 minutes
• A message is posted to Slack to notify users of the block
• During the block, IRM skips any alert group updates for that integration

Other integrations will continue posting messages until they’re also rate-limited. If another integration is blocked, it enters its own 5-minute cooldown period. Each integration is tracked and blocked independently.

After 5 minutes, the integration is unblocked and can resume posting to Slack. If it continues to exceed Slack’s limits, it will be blocked again—up to once every 5 minutes.

Impact on personal Slack notifications

Rate limits also affect personal Slack notifications:

• If an alert group can’t be posted due to rate limiting, related personal notifications in Slack are skipped.
• In some cases, even if the alert group is successfully posted, Slack may still block the personal notification due to rate limits.