Documentation Grafana Cloud Platform Workflows Reference Blueprints Declare incident for critical alerts

Declare incident for critical alerts

Declare a Grafana IRM incident automatically when a new alert group carries a severity=critical label.

This Grafana Workflows blueprint is in the Alerts category of the Blueprints library.

Before you begin

To use this blueprint, you need:

• Grafana IRM set up on your stack.
• Slack connected to Grafana IRM. For details, refer to Configure Slack for Grafana IRM.

Steps

Steps run in order from first to last. A step inside a switch branch runs only when Workflows chooses that branch.

For the inputs and outputs of each step type, refer to Step and trigger types reference.

Before you turn on the workflow, customize any steps that need input from you, such as a channel ID or a schedule name. Each step’s description says what to set.

Workflow definition

Save this definition as a YAML file to create the workflow with gcx, the Workflows API, or Terraform, or give it to a coding agent as an example. Replace FOLDER_UID with the UID of the folder for the workflow. Blueprints start disabled, so finish customizing the steps before you set enabled to true. For details, refer to Manage workflows as code.

YAML Copy

apiVersion: workflows.ext.grafana.app/v1alpha1
kind: Definition
metadata:
  name: declare-incident-for-critical-alerts
  annotations:
    grafana.app/folder: <FOLDER_UID>
spec:
  name: Declare incident for critical alerts
  enabled: false
  runOnceFor: 'declare-incident:${inputs.data.alertGroupID}'
  startWhen:
    matchingRules:
      - eventNameRegex: 'grafana_irm_app\.alertgroup\.updated'
        filter: "inputs.data.eventKind == 'created' && resources.alertgroup.event.labels.exists(l, l.key.name == 'severity' && l.value.name.matches('(?i)^critical$'))"
    schedules: []
  steps:
    - id: declare
      type: incident.create
      name: Declare the incident
      description: Creates an incident named after the alert group. Change the severity or labels to fit your process.
      inputs:
        title: '${resources.alertgroup.event.title}'
        severity: 'critical'
        labels: '[{"key": "source", "label": "alert-group"}]'
    - id: link-alert-group
      type: incident.add-note
      name: Link the alert group
      description: Adds the alert group link to the incident timeline so responders can see the alerts. No changes needed.
      inputs:
        incidentID: '${steps.declare.outputs.incidentID}'
        message: 'Declared automatically from alert group [${resources.alertgroup.event.title}](${resources.alertgroup.event.permalinks.web}) (${resources.alertgroup.event.alerts_count} alerts).'
    - id: announce
      type: slack.message.post
      name: Announce in Slack
      description: Tells your incident channel that an incident was declared. Set channelID.
      inputs:
        channelID: ''
        messageText: ':rotating_light: Declared incident *${steps.declare.outputs.incidentID}* for critical alert group <${resources.alertgroup.event.permalinks.web}|${resources.alertgroup.event.title}>.'

How it works

This section explains when the workflow runs and how its steps work together.

When it runs

This blueprint starts on the grafana_irm_app.alertgroup.updated event.

Its trigger filter starts a run only when this CEL expression is true:

text Copy

inputs.data.eventKind == 'created' && resources.alertgroup.event.labels.exists(l, l.key.name == 'severity' && l.value.name.matches('(?i)^critical$'))

It runs at most once for each run-once key. This blueprint’s key is declare-incident:${inputs.data.alertGroupID}, so a later event with the same key doesn’t start another run. For details, refer to Deduplicate workflow runs.

How the steps fit together

These notes cover how the steps connect and which Workflows behaviors the blueprint relies on.

• Triggers on grafana_irm_app.alertgroup.updated with eventKind created, which fires once when an alert group is first created.
• Workflows auto-loads the alert group from Grafana IRM as resources.alertgroup.event, with title, state, alerts_count, permalinks.web, and labels, where each label is {key: {name}, value: {name}}.
• Trigger filters run after resources load, so the filter uses the CEL exists() macro on resources.alertgroup.event.labels to look for a severity label equal to critical.
• inputs.data holds what the event is about (eventKind, alertGroupID).
• runOnceFor keys on the alert group ID.
• incident.create returns the new incidentID, which later steps use to add a note with the alert group link (resources.alertgroup.event.permalinks.web).
• Customize the label key and value in the trigger filter, the incident severity and labels, and the Slack channel.

Next steps

• Blueprints
• Start from a blueprint
• Manage workflows as code
• Step and trigger types reference