Slack integration for Grafana IRM
Grafana Cloud RSS

Slack integration for Grafana IRM

Note

The Grafana IRM Slack integration combines the features of the OnCall and Incident Slack apps into a single, unified Grafana IRM Slack app.

If you have an existing installation of the legacy Slack integrations, do not uninstall them. Refer to the migration instructions for more information.

The Grafana IRM Slack integration brings your incident response and on-call workflows directly into your Slack workspace, helping your team focus on alert resolution with less friction.

Key features and benefits

Integrating your Slack workspace allows users and teams to be notified of alerts directly in Slack with automated alert escalation steps and user notification preferences. Users can take a number of alert actions directly from Slack, including acknowledging and resolving alert groups, adding resolution notes, and more.

Refer to the following sections to learn more:

When it comes to incidents, users can declare, collaborate on, and manage incident-worthy events without leaving Slack. They can also automatically create incident-specific channels, track the timeline of events, interact with incidents via Slash commands, and more.

Refer to the following sections to learn more:

Before you begin

To install the Grafana IRM Slack app, you must:

  • Have an Admin role in Grafana IRM
  • Be a Slack workspace Admin or Owner
  • Allow Grafana IRM to access your Slack workspace

After the IRM Slack app has been added to your Slack workspace, Admins and Editors can configure escalation chains and other notifications to route to designated Slack channels.

In order to use the Slack slash command /grafana, you must be an IRM user with a linked Slack account and necessary permissions.

To learn more, refer to Connect your Slack user to Grafana IRM.

Install the Grafana IRM Slack integration

The Grafana IRM Slack integration is installed and managed from the unified IRM app in Grafana Cloud. Installing the integration enables Slack notifications and incident collaboration features across all IRM workflows.

Follow these steps to install the Slack integration:

  1. In Grafana Cloud, open the main menu and navigate to Alerts & IRM > IRM > Integrations.
  2. Click the Apps tab and select the Slack IRM integration.
  3. Click Install integration.
  4. After being redirected to the Slack connection page, verify the Slack workspace is correct or Add another workspace.
  5. If needed, provide your Slack workspace URL and sign in with your Slack credentials.
  6. Follow the Slack prompts to review permissions and specify a default channel.
  7. Click Allow to give Grafana IRM permission to access your Slack workspace.

If you already have the legacy OnCall or Incident Slack integration installed, use the Migration guide to migrate your existing Slack configurations.

For more information about the required permissions, refer to the Permissions section.

Connect your Slack user to Grafana IRM

For users to gain full access to Grafana IRM functionality in Slack, follow these steps to map your Grafana IRM user account to your Slack user account:

  1. In Grafana Cloud, click your avatar and go to your Profile.
  2. Select the IRM tab.
  3. In the Slack connection section, click Open Slack connection page.
  4. Authorize your Slack account and follow the prompts to verify and link your Slack account.

After linking your Slack account, you can start using Grafana IRM in Slack in accordance with the role and permissions assigned to you in Grafana. The actions you can take in Slack depend on your permissions in Grafana IRM.

For more information on roles and permissions, refer to the Grafana Cloud user roles and permissions documentation.

Configure channels and notifications

Grafana IRM provides flexible configuration options to tailor Slack notifications to your team’s needs. This section outlines how to set up escalation chains, user notifications, on-call notifications, and incident channel settings within Slack.

Configure Escalation Chains with Slack notifications

After setting up your Slack integration, you can configure escalation chains to send notifications via Slack for firing alerts in Grafana IRM.

There are two Slack notification methods that can be integrated into escalation chains:

  • Notify all members of a Slack channel
  • Notify a specific Slack user group

To configure these in your escalation chains:

  1. In Grafana Cloud, navigate to Alerts & IRM > IRM > Escalation chains.
  2. Select an existing escalation chain or create a new one by clicking + Add escalation chain.
  3. Use the dropdown under Add escalation step to choose and configure your Slack notification preferences.

Configure user notification rules with Slack mentions

Before configuring Slack notifications, ensure you have connected your Slack account as described in Connect notification channels.

To configure Slack mentions for your notification rules:

  1. Go to your Grafana Cloud Profile.
  2. Select the IRM tab.
  3. Click Notification rules.
  4. Click + Add notification rule.
  5. Choose Slack mentions from the notification channel dropdown.
  6. Configure the remaining notification rule settings as needed.

If you don’t see Slack as an available notification channel in your user profile, it may be disabled by your organization administrator. For more information about notification channel visibility and management, refer to the Notification channel documentation.

Configure on-call shift notifications in Slack

The IRM Slack integration also supports automated notifications for on-call shifts, helping teams stay informed of their duties. Admins can set up these notifications in the Notification Preferences section:

  1. When an on-call shift notification is sent to a person or channel, click the gear icon to open Notifications preferences.
  2. Configure the notification behavior for future shifts according to your team’s preferences.

Configure Incident Slack channels

Configure the following incident-specific Slack channel settings to suit your organization’s process and preferences.

Incident channel creation

Customize when incident Slack channels are created to align with your organization’s workflows.

To configure incident channel creation:

  1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack IRM.
  2. Specify the following settings:
    • Create Slack channel: Determine when to automatically create a Slack channel for an incident.
    • Post updates to channel: Choose whether to automatically post incident updates to the created Slack channel.
    • Invite users: Specify if users with an active role should be automatically invited to the incident channel.

Automatic channel archive

To maintain good Slack workspace hygiene, you can automatically archive resolved incident channels after a configurable number of days.

Consider a timeframe of 30-60 days to allow time for PIR, follow-ups, and stakeholder visibility.

To configure automatic channel archive:

  1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack IRM.
  2. Set the desired number of days after incident resolution for channels to automatically archive.
  3. Click Save settings.

Customize Slack channel prefixes

You can define custom channel prefixes to organize and identify incident channels more effectively.

To customize Slack channel prefixes:

  1. In Grafana Cloud, navigate to Alerts & IRM > IRM > Settings.
  2. Select the Incidents tab and find the Prefixes section.
  3. Click + Add Prefix to create a new prefix.
  4. Enter a Name and Description for the prefix.
  5. Edit or delete existing prefixes as needed.
  6. When declaring a new incident, select the desired prefix from the dropdown menu.

For more information, refer to the Customize incident channel prefixes documentation.

Configure Incident announcements

Incident announcements help keep your team aware and informed during critical events. Configure these announcements to ensure that stakeholders are kept up-to-date in Slack during an incident.

To configure incident announcements:

  1. In Grafana Cloud, navigate to IRM > Integrations > Apps and select Slack IRM.
  2. Select the Slack channel where you want to send notifications. You can either choose from the dropdown menu or manually add your Channel ID. The Channel ID can be found in the About tab of the Slack channel.
  3. Define the incident-specific fields:
    • Incident type: Choose whether the incident is internal or private.
    • Include incidents: Specify which types of incidents to announce—options include all incidents, drills, or non-drills.
  4. Apply filters to your incident announcements to tailor the notifications to specific channels:
    • Incident filter: For example, you might filter by label, such as label: 'squad:datasources'.

Manage Slack attachments

When you use the 🤖 emoji reaction on a Slack message containing a file, the file is securely copied to Grafana Cloud storage. This ensures your incident timeline remains intact, even if the file is later deleted from Slack.

  • File retention: Control attachment retention in your incident timeline. Removing the 🤖 reaction deletes the attached files from Grafana Cloud storage.
  • Incident web app: Deleting an item from the timeline within the IRM web app also removes the associated file from Grafana Cloud storage.
  • File size limit: Files in Grafana IRM are limited to 100MB. Files too large to persist are still accessible via a link to the source file in Slack.

Available Slack commands

Note

The /escalate and /incident Slack commands have been deprecated. Use /grafana to learn more.

The /grafana Slack commands allow users and teams to respond to alert groups and collaborate on incidents directly from Slack.

Tip

Use the 🤖 robot emoji Slack reaction to add important messages to the incident timeline.

CommandDescription
/grafanaList of all /grafana commands
/grafana stacksList all available stacks
/grafana set-stackSet your default stack
/grafana escalatePage a user or a team
/grafana incidentPrefix for Incident-related commands
CommandDescription
/grafana incident newCreate new incident
/grafana incident new 'title'Create new incident with severity
/grafana incident listList all active incidents
/grafana incident rolesFind out who’s involved
/grafana incident talkSet up a collaboration space
/grafana incident statusGet a live inline update
/grafana incident severitySet the incident severity
/grafana incident severity majorUpdate the incident severity to major
/grafana incident tasksCreate, assign and manage tasks
/grafana incident tasks helpMore information about tasks
/grafana incident task add 'deploy new release'Add a new task
/grafana incident task add 'deploy new release' @bobAdd a new task and assign to @bob
/grafana incident tasks listView current tasks
/grafana incident notesAdd and view incident notes
/grafana incident notes helpMore information about notes
/grafana incident note add 'customers notified'Add a new note
/grafana incident notes listView current notes

Permissions scope and purpose

The Grafana IRM Slack app only requests permissions that are essential to its proper function and integration with Slack. Refer to the Slack documentation for more information on permission scopes.

By granting access to the app, you are authorizing Grafana IRM the following permissions in your Slack workspace:

Workspace and user access

PermissionDescriptionPurpose
team:readView the workspace name, email domain, and iconUsed for identification and to establish an association with your workspace
users:readView people in a workspaceUsed to find users by email and facilitate account matching between Grafana and your workspace
users.profile:readView profile details about people in a workspaceUsed to fetch profile data, such as Slack handles and avatars
users:writeSet presence for Grafana IRM Slack appAllows @GrafanaIRM to be added to your workspace
usergroups:readView user groups in a workspaceRequired to connect on-call schedules and escalation chains to Slack user groups
usergroups:writeCreate and manage user groupsRequired to connect on-call schedule and escalation chains to Slack user groups
incoming-webhookCreate one-way webhooks to post messages to a specific channelUsed to display a channel picker in the installation sequence

Public channel access

PermissionDescriptionPurpose
channels:readView basic information about public channelsUsed when adding a new channel to an escalation chain and to locate and update incident-specific channels
channels:manageManage public channels that Grafana IRM has been added to and create new onesAllows the app to create new channels and invite users to these channels
channels:historyView messages and other content in public channels that Grafana IRM has been added toAllows @GrafanaIRM to list messages in channels where the app has access to be included in the resolution notes
bookmarks:readList bookmarksUsed to access bookmarks, such as PIR documents and Google Meet links, related to incidents
bookmarks:writeCreate, edit, and remove bookmarksNecessary for managing incident-related bookmarks, including PIR documents and Google Meet links
files:readView files shared in channels and conversations that Grafana IRM is a part ofFor incident-related file sharing and collaboration within authorized channels
files:writeUpload, edit, and delete files as Grafana IRMUsed to upload image attachments so they appear in the Incident timeline
pins:readView pinned content in channels and conversations that Grafana Incident has been added toUsed to access and display pinned content related to incidents

Private channel access

PermissionDescriptionPurpose
groups:readView basic information about private channels that Grafana IRM has been added toUsed to access information from private incident-specific channels
groups:writeManage private channels that Grafana IRM has been added to and create new onesRequired to create private incident-specific channels for private incidents
groups:historyList messages from private incident-specific channelsAllows @GrafanaIRM to list messages from private incident channels to be included in the resolution notes
usergroups:writeCreate and manage user groupsRequired for connecting on-call schedules to Slack user groups
usergroups:readView user groups in a workspaceRequired for connecting on-call schedules to Slack user groups

Message and conversation access

PermissionDescriptionPurpose
app_mentions:readView messages and other content in public channels that Grafana IRM is a part ofEnables the app to read messages and related content within authorized channels
chat:writePost messages in approved channels and conversationsAllows @GrafanaIRM to post messages to Slack
chat:write.customizeSend messages as Grafana IRM with a customized username and avatarAllows @GrafanaIRM to post messages to Slack
chat:write.publicSend messages to channels Grafana IRM isn’t a member ofAllows @GrafanaIRM to post messages to Slack
reaction:readView emoji reactions and their associated content in channels and conversations that Grafana IRM has been added toAllows @GrafanaIRM to monitor message events for the 🤖 emoji to be included in the incident timeline
reaction:writeAdd and edit emoji reactionsRequired to include messages with the 📝 emoji on resolution notes
im:readView basic information about direct messages that Grafana IRM has been added toEnables the app to send alert group notifications to users via direct message
im:writeStart direct messages with peopleUsed to notify users about alerts via direct message as well as invite users who create an incident to the incident channel
im:historyView messages and other content in direct messages that Grafana IRM has been added toAllows @GrafanaIRM to monitor message events in direct messages that it’s a part of
mpim:historyView messages and other content in group direct messages that Grafana IRM has been added toAllows @GrafanaIRM to monitor message events in group direct messages that it’s a part of