Major incident kickoff
Note
Grafana Workflows is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.
Spin up a major incident in seconds. Creates the incident Slack channel, assigns the on-call incident commander, posts a help message, and bookmarks the runbook.
This Grafana Workflows blueprint is in the Incidents category of the Blueprints library.
Before you begin
To use this blueprint, you need:
- Grafana IRM set up on your stack.
- Slack connected to Grafana IRM. For details, refer to Configure Slack for Grafana IRM.
Steps
Steps run in order from first to last. A step inside a switch branch runs only when Workflows chooses that branch.
For the inputs and outputs of each step type, refer to Step and trigger types reference.
Before you turn on the workflow, customize any steps that need input from you, such as a channel ID or a schedule name. Each step’s description says what to set.
Workflow definition
Save this definition as a YAML file to create the workflow with gcx, the Workflows API, or Terraform, or give it to a coding agent as an example.
Replace FOLDER_UID with the UID of the folder for the workflow.
Blueprints start disabled, so finish customizing the steps before you set enabled to true.
For details, refer to Manage workflows as code.
apiVersion: workflows.ext.grafana.app/v1alpha1
kind: Definition
metadata:
name: major-incident-kickoff
annotations:
grafana.app/folder: <FOLDER_UID>
spec:
name: Major incident kickoff
enabled: false
runOnceFor: 'major-incident:${inputs.data.incidentID}'
startWhen:
matchingRules:
- eventNameRegex: 'grafana_irm_app\.incident\.updated'
filter: "resources.incident.event.status == 'active' && resources.incident.event.severity.matches('(?i)^critical$')"
schedules: []
steps:
- id: create-channel
type: incident.slack-channel.create
name: Create incident channel
description: >-
Creates a Slack channel linked to the incident and invites everyone who already holds an
incident role. IRM posts incident activity to the channel. Change channelName to match
your naming convention. Slack naming rules are applied automatically.
inputs:
incidentID: '${inputs.data.incidentID}'
channelName: 'incident-${inputs.data.incidentID}'
postUpdates: 'true'
inviteUsers: 'true'
- id: get-on-call
type: irm.schedule-current-oncall
name: Find the on-call commander
description: >-
Looks up who is currently on call in your incident commander rotation. Set
scheduleIdentifier to the exact name of the IRM schedule, or switch identifierType to id
and paste the schedule ID.
inputs:
scheduleIdentifier: ''
identifierType: 'name'
- id: assign-commander
type: switch
name: Assign commander if someone is on call
description: >-
Assigns the first on-call user as incident commander. When nobody is on call, the
default branch leaves a note on the timeline so the team assigns one manually.
branches:
- condition: "steps['get-on-call'].outputs.count > 0"
steps:
- id: add-commander
type: incident.add-participant
name: Make on-call user the commander
description: Gives the on-call user the commander role on the incident. No changes needed.
inputs:
incidentID: '${inputs.data.incidentID}'
userID: "${steps['get-on-call'].outputs.users[0].email}"
role: 'commander'
- steps:
- id: note-no-commander
type: incident.add-note
name: Note that no commander was assigned
description: Records that nobody was on call so the team knows to pick a commander. No changes needed.
inputs:
incidentID: '${inputs.data.incidentID}'
message: 'Nobody was on call in the incident commander schedule, so no commander was assigned automatically. Assign one from the incident page.'
- id: post-help
type: slack.message.post
name: Post help message
description: >-
Posts a help message to the new channel that explains roles and how to work the incident. Edit the text to match your major incident process.
inputs:
channelID: "${steps['create-channel'].outputs.channelID}"
messageText: |
:rotating_light: *Major incident declared: ${resources.incident.event.title}*
*Commander:* ${steps['get-on-call'].outputs.count > 0 ? steps['get-on-call'].outputs.users[0].username : 'not assigned yet, pick one now'}
*Severity:* ${resources.incident.event.severity}
*Incident page:* ${resources.incident.event.overviewURL}
*How we work this incident*
• The commander coordinates. Ask before you take any risky action.
• Keep the discussion in this channel. Use threads for side investigations.
• Post key updates on the incident page so stakeholders stay informed.
• Follow the runbook bookmarked at the top of this channel.
- id: bookmark-runbook
type: slack.channel.add-bookmark
name: Bookmark the runbook
description: Adds your major incident runbook to the channel bookmarks. Set bookmarkUrl to your runbook link.
inputs:
channelID: "${steps['create-channel'].outputs.channelID}"
bookmarkTitle: 'Major incident runbook'
bookmarkUrl: ''
bookmarkEmoji: ':book:'
- id: bookmark-incident
type: slack.channel.add-bookmark
name: Bookmark the incident page
description: Adds the Grafana IRM incident page to the channel bookmarks. No changes needed.
inputs:
channelID: "${steps['create-channel'].outputs.channelID}"
bookmarkTitle: 'Incident page'
bookmarkUrl: '${resources.incident.event.overviewURL}'
bookmarkEmoji: ':fire:'
- id: note-kickoff
type: incident.add-note
name: Record the kickoff on the timeline
description: Leaves a timeline note that links the new channel. No changes needed.
inputs:
incidentID: '${inputs.data.incidentID}'
message: "Major incident kickoff ran. Channel: #${steps['create-channel'].outputs.channelName}"How it works
This section explains when the workflow runs and how its steps work together.
When it runs
This blueprint starts on the grafana_irm_app.incident.updated event.
Its trigger filter starts a run only when this CEL expression is true:
resources.incident.event.status == 'active' && resources.incident.event.severity.matches('(?i)^critical$')It runs at most once for each run-once key. This blueprint’s key is major-incident:${inputs.data.incidentID}, so a later event with the same key doesn’t start another run. For details, refer to Deduplicate workflow runs.
How the steps fit together
These notes cover how the steps connect and which Workflows behaviors the blueprint relies on.
- Runs once per incident when an active incident is Critical, whether it was declared Critical or raised to Critical later.
- Incident events carry only IDs, so the trigger filter reads the auto-loaded incident at
resources.incident.event(trigger filters run after resources load). runOnceForkeys on the incident ID.- Because the severity check lives in the trigger filter rather than in a filter step, events that do not match release the run-once key, so a Pending incident that is later raised to Critical still fires exactly once.
- Steps: create a Slack channel linked to the incident (
incident.slack-channel.create, so IRM posts updates there), look up who is on call in the incident commander schedule, assign that person the commander role when someone is on call (switch with a default branch that leaves a timeline note when nobody is), post a help message, bookmark the runbook and the incident page, and record a timeline note. - Steps after the switch reference get-on-call, not the branch steps, because branch step outputs are not visible outside their branch.
- Customize the schedule name, the runbook URL, the severity regex in the trigger filter, and the help message.
- If your IRM Slack integration already creates a channel for every incident, delete create-channel and use
resources.incident.event.slackChannelIDinstead.
Next steps
Was this page helpful?
Related resources from Grafana Labs
