Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Platformbreadcrumb arrow Workflowsbreadcrumb arrow Referencebreadcrumb arrow Blueprintsbreadcrumb arrow Annotate dashboards with incidents
Public preview Grafana Cloud
Last reviewed: November 9, 2026

Annotate dashboards with incidents

Note

Grafana Workflows is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

When an incident is resolved, add an annotation that spans the incident to your Grafana dashboards, so it lines up with your metrics.

This Grafana Workflows blueprint is in the Integrations category of the Blueprints library.

Before you begin

To use this blueprint, you need:

  • Grafana IRM set up on your stack.
  • A workflow secret named grafana-api-token.

To create a secret, refer to Manage secrets.

Steps

Steps run in order from first to last. A step inside a switch branch runs only when Workflows chooses that branch.

StepIDTypeDescription
Create the annotationcreate-annotationhttp.callCreates a region annotation from incident start to resolution, tagged with incident and the severity. Replace your-stack.grafana.net with your Grafana Cloud stack, for example mycompany.grafana.net. Also create a workflow secret named grafana-api-token that holds a service account token with permission to write annotations.
Fail if the API call failedcheck-responsefilterFails the run with an error when Grafana does not accept the annotation. No changes needed.
Record the annotationrecord-annotationincident.add-noteNotes the annotation ID on the incident timeline. No changes needed.

For the inputs and outputs of each step type, refer to Step and trigger types reference.

Before you turn on the workflow, customize any steps that need input from you, such as a channel ID or a schedule name. Each step’s description says what to set.

Workflow definition

Save this definition as a YAML file to create the workflow with gcx, the Workflows API, or Terraform, or give it to a coding agent as an example. Replace FOLDER_UID with the UID of the folder for the workflow. Blueprints start disabled, so finish customizing the steps before you set enabled to true. For details, refer to Manage workflows as code.

YAML 
apiVersion: workflows.ext.grafana.app/v1alpha1
kind: Definition
metadata:
  name: annotate-dashboards-with-incidents
  annotations:
    grafana.app/folder: <FOLDER_UID>
spec:
  name: Annotate dashboards with incidents
  enabled: false
  runOnceFor: 'annotate:${inputs.data.incidentID}'
  startWhen:
    matchingRules:
      - eventNameRegex: 'grafana_irm_app\.incident\.updated'
        filter: "inputs.data.eventKind == 'incident_updated_status' && inputs.data.newStatus == 'resolved' && !resources.incident.event.isDrill"
    schedules: []
  resources:
    secrets:
      grafanaApiToken: 'secret://grafana-api-token'
  steps:
    - id: create-annotation
      type: http.call
      name: Create the annotation
      description: >-
        Creates a region annotation from incident start to resolution, tagged with incident and the
        severity. Replace your-stack.grafana.net with your Grafana Cloud stack, for example
        mycompany.grafana.net. Also create a workflow secret named grafana-api-token that holds a
        service account token with permission to write annotations.
      inputs:
        method: 'POST'
        url: 'https://your-stack.grafana.net/api/annotations'
        body: |
          {
            "time": ${int(timestamp(resources.incident.event.incidentStart != '' ? resources.incident.event.incidentStart : resources.incident.event.createdTime)) * 1000},
            "timeEnd": ${int(timestamp(resources.incident.event.closedTime)) * 1000},
            "tags": [
              "incident",
              ${json_stringify(resources.incident.event.severity)}
            ],
            "text": ${json_stringify('Incident ' + inputs.data.incidentID + ': ' + resources.incident.event.title)}
          }
        authMethod: 'bearer'
        authSecret: '${resources.secret.grafanaApiToken}'
    - id: check-response
      type: filter
      name: Fail if the API call failed
      description: Fails the run with an error when Grafana does not accept the annotation. No changes needed.
      inputs:
        condition: "steps['create-annotation'].outputs.responseCode == 200"
        failOnExit: 'true'
    - id: record-annotation
      type: incident.add-note
      name: Record the annotation
      description: Notes the annotation ID on the incident timeline. No changes needed.
      inputs:
        incidentID: '${inputs.data.incidentID}'
        message: "Added Grafana annotation ${string(json_parse(steps['create-annotation'].outputs.responseBody).id)} for this incident."

How it works

This section explains when the workflow runs and how its steps work together.

When it runs

This blueprint starts on the grafana_irm_app.incident.updated event.

Its trigger filter starts a run only when this CEL expression is true:

text 
inputs.data.eventKind == 'incident_updated_status' && inputs.data.newStatus == 'resolved' && !resources.incident.event.isDrill

It runs at most once for each run-once key. This blueprint’s key is annotate:${inputs.data.incidentID}, so a later event with the same key doesn’t start another run. For details, refer to Deduplicate workflow runs.

How the steps fit together

These notes cover how the steps connect and which Workflows behaviors the blueprint relies on.

  • Triggers on eventKind incident_updated_status when newStatus is resolved.
  • runOnceFor keys on the incident ID.
  • http.call POSTs to the Grafana HTTP API (/api/annotations) with a service account token stored as a workflow secret and referenced only in authSecret.
  • The body is a JSON string built with CEL.
  • timestamp() parses the RFC 3339 incident times, int() converts a timestamp to Unix seconds, and multiplying by 1000 gives the milliseconds the annotations API expects.
  • Text values go through json_stringify() to stay valid JSON.
  • The annotation has no dashboardUID, so it is an organization annotation.
  • Show it on any dashboard with an annotation query that filters on the tag incident.
  • A filter with failOnExit makes API errors fail the run.
  • The last step parses the response with json_parse and records the annotation ID on the incident timeline.
  • Customize the stack host in the URL, the secret name, and the tags.

Next steps