Easily page participants to accelerate incident response in Grafana IRM

Incidents almost never happen in a vacuum. When you receive an alert about a potential issue, odds are pretty good that you’ll need to navigate between different tools and teams to get things resolved. Of course, timing is critical in these situations, so the easier it is to communicate — between both tools and teams — the better off you’ll be.

That was the inspiration for a new capability in Grafana Incident Response & Management (Grafana IRM) that ties Grafana OnCall and Grafana Incident closer together. With this new functionality, you can now use either service to directly notify other users or teams so you can more easily pull them into your incident response.

Let’s take a closer look at this new capability, how it can benefit your incident response team, and how to start using it today.

A more seamless IRM experience

For many incident response teams, pulling in other groups can be a manual process that, while necessary, can get in the way of actually addressing the problem. Not only do you have to track down which team you need assistance from, you also have to find out who on that team is available and which communication channel they’re most likely to see first. If you don’t have the proper escalation checks in place, or if your teams are too siloed, you also run the risk of not hearing back in a timely manner. 

With this new capability, we want to streamline the process of connecting workflows in Grafana IRM so it reduces toil and gives all the necessary parties proper visibility into the incident. The ability to add participants to an incident utilizes existing functionality in Grafana OnCall, where every user can define their personal notification policy so others know how (Slack, email, mobile, etc.) and when that team members can be notified about an incident. 

With those settings in place, you can now go into Grafana OnCall or Grafana Incident and quickly select individuals or teams so they’re notified about an escalation. You can also include the severity of the incident and notify them in the way that works best for them. Moreover, if they don’t respond, it will trigger an escalation chain so the notification doesn’t end up in the IRM abyss. 

We previously had scheduling-based functionality in Grafana OnCall that could be used for the same purpose, but this simplifies the process considerably, making it easier to get the attention of others when you need to.This is an entirely new capability in Grafana Incident, and it’s part of our commitment to improving the way users connect their workflows in Grafana IRM.

How it works

Adding participants is easy with this new feature. Just follow these steps:

• Log into your Grafana Cloud instance. If you don’t have a Grafana Cloud account, you can sign up for a forever-free account today.

From here, you can add a participant through either Grafana OnCall or Grafana Incident. Next, we’ll show you both options, starting with Grafana OnCall.

From Grafana OnCall

• From the Alert Groups page, click the + Escalation button. A drawer should appear from the right hand side with more details.

• Provide a brief description about why you are paging the team/user(s) in the “What is going on?” input. This helps provide the users being paged with some basic context.

• Next, select which team/user(s) you would like to page. Click + Invite in the “Participants” section. You must select at least one team or one user. You can only select one team, as this team’s Direct Paging integration is what is used to create the associated Alert Group. (The UI also shows the number of on-call users currently available.) If you do not select a team, a generic “No team” Direct Paging integration will be used to generate the associated Alert Group.

  • On the “Teams” tab you can search by team name
  • On the “Users” tab you can search by username, email, or the name of a team the user belongs to.

• If you’ve selected user(s), you can specify which type of personal notification policy you would like to use. (Note: If a selected user does not have their personal notification policy configured, we send them an email containing the direct paging information.)

• Click Create. You should be redirected to the associated Alert Group’s detail page.

• From here you can see details about each user that has been paged. Additionally, you can add or remove Participants from the Alert Group (so long as the Alert Group has not been resolved).

• Additionally, the above workflow can be done from within Slack. Use the /escalate slash command from within Slack to get started:

For more information, please refer to the Grafana OnCall documentation. 

From Grafana Incident

• Open an incident details page.

• Click on + Invite on the Participants sidebar and select either a user or a team to be paged.

• On the Confirm invitation modal, click Confirm to send the notification, which will go directly to the participant’s preferred communication tool.

• When the paged participant opens the incident page, they will be prompted to accept or decline the invitation. This will automatically resolve the related alert group and log that event onto the incident timeline.

For more information, read the Grafana Incident documentation. 

Add participants today for smoother incident response

There aren’t many solutions on the market that include both on-call and incident response capabilities in one package, so we’re excited for our Grafana IRM users to start implementing this new capability into their workflows. Incident response can be a very stressful experience, so we’re always looking for ways to make the process as smooth as possible. In fact, we continue to work toward a more seamless experience across the Grafana IRM suite of tools — Grafana Alerting, Grafana Incident, Grafana OnCall, and Grafana SLO — so stay tuned for more updates like this in the coming months.

For now, if you have any questions or want to let us know what you think about this new functionality, don’t hesitate to reach out to your Grafana Cloud account representative today. And if you don’t have one, you can get started with a forever-free account today!

Grafana Cloud is the easiest way to get started with metrics, logs, traces, dashboards, and more. We have a generous forever-free tier and plans for every use case. Sign up for free now!