--- title: "Annotate dashboards with incidents | Grafana Cloud documentation" description: "When an incident is resolved, add an annotation that spans the incident to your Grafana dashboards, so it lines up with your metrics." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Annotate dashboards with incidents > Note > > Grafana Workflows is currently in [public preview](https://grafana.com/docs/release-life-cycle/). Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available. When an incident is resolved, add an annotation that spans the incident to your Grafana dashboards, so it lines up with your metrics. This Grafana Workflows blueprint is in the **Integrations** category of the Blueprints library. ## Before you begin To use this blueprint, you need: - Grafana IRM set up on your stack. - A workflow secret named `grafana-api-token`. To create a secret, refer to [Manage secrets](https://grafana.com/docs/grafana-cloud/platform/workflows/manage/secrets). ## Steps Steps run in order from first to last. A step inside a switch branch runs only when Workflows chooses that branch. Expand table | Step | ID | Type | Description | |-----------------------------|---------------------|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Create the annotation | `create-annotation` | `http.call` | Creates a region annotation from incident start to resolution, tagged with incident and the severity. Replace `your-stack.grafana.net` with your Grafana Cloud stack, for example `mycompany.grafana.net`. Also create a workflow secret named grafana-api-token that holds a service account token with permission to write annotations. | | Fail if the API call failed | `check-response` | `filter` | Fails the run with an error when Grafana does not accept the annotation. No changes needed. | | Record the annotation | `record-annotation` | `incident.add-note` | Notes the annotation ID on the incident timeline. No changes needed. | For the inputs and outputs of each step type, refer to [Step and trigger types reference](https://grafana.com/docs/grafana-cloud/platform/workflows/reference/step-trigger-types). Before you turn on the workflow, customize any steps that need input from you, such as a channel ID or a schedule name. Each step’s description says what to set. ## Workflow definition Save this definition as a YAML file to create the workflow with gcx, the Workflows API, or Terraform, or give it to a coding agent as an example. Replace *FOLDER\_UID* with the UID of the folder for the workflow. Blueprints start disabled, so finish customizing the steps before you set `enabled` to `true`. For details, refer to [Manage workflows as code](https://grafana.com/docs/grafana-cloud/platform/workflows/as-code). YAML ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```yaml apiVersion: workflows.ext.grafana.app/v1alpha1 kind: Definition metadata: name: annotate-dashboards-with-incidents annotations: grafana.app/folder: spec: name: Annotate dashboards with incidents enabled: false runOnceFor: 'annotate:${inputs.data.incidentID}' startWhen: matchingRules: - eventNameRegex: 'grafana_irm_app\.incident\.updated' filter: "inputs.data.eventKind == 'incident_updated_status' && inputs.data.newStatus == 'resolved' && !resources.incident.event.isDrill" schedules: [] resources: secrets: grafanaApiToken: 'secret://grafana-api-token' steps: - id: create-annotation type: http.call name: Create the annotation description: >- Creates a region annotation from incident start to resolution, tagged with incident and the severity. Replace your-stack.grafana.net with your Grafana Cloud stack, for example mycompany.grafana.net. Also create a workflow secret named grafana-api-token that holds a service account token with permission to write annotations. inputs: method: 'POST' url: 'https://your-stack.grafana.net/api/annotations' body: | { "time": ${int(timestamp(resources.incident.event.incidentStart != '' ? resources.incident.event.incidentStart : resources.incident.event.createdTime)) * 1000}, "timeEnd": ${int(timestamp(resources.incident.event.closedTime)) * 1000}, "tags": [ "incident", ${json_stringify(resources.incident.event.severity)} ], "text": ${json_stringify('Incident ' + inputs.data.incidentID + ': ' + resources.incident.event.title)} } authMethod: 'bearer' authSecret: '${resources.secret.grafanaApiToken}' - id: check-response type: filter name: Fail if the API call failed description: Fails the run with an error when Grafana does not accept the annotation. No changes needed. inputs: condition: "steps['create-annotation'].outputs.responseCode == 200" failOnExit: 'true' - id: record-annotation type: incident.add-note name: Record the annotation description: Notes the annotation ID on the incident timeline. No changes needed. inputs: incidentID: '${inputs.data.incidentID}' message: "Added Grafana annotation ${string(json_parse(steps['create-annotation'].outputs.responseBody).id)} for this incident." ``` ## How it works This section explains when the workflow runs and how its steps work together. ### When it runs This blueprint starts on the `grafana_irm_app.incident.updated` event. Its trigger filter starts a run only when this CEL expression is true: text ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```text inputs.data.eventKind == 'incident_updated_status' && inputs.data.newStatus == 'resolved' && !resources.incident.event.isDrill ``` It runs at most once for each run-once key. This blueprint’s key is `annotate:${inputs.data.incidentID}`, so a later event with the same key doesn’t start another run. For details, refer to [Deduplicate workflow runs](https://grafana.com/docs/grafana-cloud/platform/workflows/build/triggers/#deduplicate-workflow-runs). ### How the steps fit together These notes cover how the steps connect and which Workflows behaviors the blueprint relies on. - Triggers on `eventKind` `incident_updated_status` when `newStatus` is resolved. - `runOnceFor` keys on the incident ID. - `http.call` POSTs to the Grafana HTTP API (`/api/annotations`) with a service account token stored as a workflow secret and referenced only in `authSecret`. - The body is a JSON string built with CEL. - `timestamp()` parses the RFC 3339 incident times, `int()` converts a timestamp to Unix seconds, and multiplying by 1000 gives the milliseconds the annotations API expects. - Text values go through `json_stringify()` to stay valid JSON. - The annotation has no `dashboardUID`, so it is an organization annotation. - Show it on any dashboard with an annotation query that filters on the tag incident. - A filter with `failOnExit` makes API errors fail the run. - The last step parses the response with `json_parse` and records the annotation ID on the incident timeline. - Customize the stack host in the URL, the secret name, and the tags. ## Next steps - [Blueprints](https://grafana.com/docs/grafana-cloud/platform/workflows/reference/blueprints) - [Start from a blueprint](https://grafana.com/docs/grafana-cloud/platform/workflows/build/use-blueprints) - [Manage workflows as code](https://grafana.com/docs/grafana-cloud/platform/workflows/as-code) - [Step and trigger types reference](https://grafana.com/docs/grafana-cloud/platform/workflows/reference/step-trigger-types)