Documentation Grafana Cloud Alerts and IRM Workflows Troubleshoot

Troubleshoot workflows

Use this page to diagnose the most common Workflows failures. Each section names a symptom, explains why it happens, and suggests where to look in the editor.

Step failure model

Workflows stop on the first error. When any step returns an error, the workflow run is marked Failed and no later steps run. Branches inside a switch step inherit the same behavior: if a step inside the executing branch fails, the run halts.

There is no per-step retry policy in the workflow definition schema today. To recover from transient failures, run the workflow again with the same event payload. For details on how to capture and resend a payload, refer to Observe workflow runs.

The whole workflow has a 5 minute execution timeout. Plan your wait steps and HTTP calls accordingly.

My workflow didn’t run

A workflow that doesn’t run usually means the engine never matched the event to your workflow.

Check, in order:

1. The workflow is enabled. In the editor header, confirm the Enable toggle is on. Disabled workflows do not run on incoming events. They can still be triggered manually.
2. At least one matching rule exists. In the editor’s Triggers section or the YAML startWhen.matchingRules, confirm there’s at least one eventNameRegex. A workflow with no matching rules and no schedules cannot fire.
3. The regex matches the stripped event name. The engine matches against {app}.{resource}.{event}, not {app}.{instance-id}.{resource}.{event}. A pattern like grafana_irm_app\.42\.incident\.updated never matches because the engine strips 42 before testing. For details, refer to Configure triggers.
4. The event name uses the correct app and resource tokens. Use grafana_irm_app (with underscores), not grafana-irm-app. Use alertgroup (singular), not alert_groups. Refer to Sample event payloads for the complete list.
5. The workflow’s namespace matches the event’s stack. Workflows only fire for events from their own Grafana Cloud stack.

If the workflow still doesn’t run, trigger it manually with Run to confirm the steps work. If manual runs succeed, the matching rule is the problem.

My CEL expression evaluates to empty

CEL expressions that reference missing fields evaluate to empty strings. This is most common when the expression’s path doesn’t match the actual event payload.

Check, in order:

1. Inspect the actual event shape. Open Latest runs, select a recent run, and use View details on the first step to see the input the engine produced. Compare the keys you see to the keys in your expression.
2. Use the right root object. Use inputs.data.* for fields from the triggering event, steps.<step-id>.outputs.<field> for outputs from earlier steps, and resources.incident.event.* or resources.alertgroup.event.* for auto-loaded resources.
3. Quote hyphenated step IDs with brackets. Use steps["my-step"].outputs.result instead of steps.my-step.outputs.result.
4. Escape literal ${. Use $${ to emit a literal ${ in templated strings.

For details, refer to Use CEL expressions.

My secret returned 401 or “secret not found”

Secrets failures occur during step execution, not at validation time.

Check, in order:

1. The secret reference uses the right URI. Confirm the URI in your workflow’s resources.secrets map matches a secret you created in the Workflows app’s Secrets page or in Grafana Secrets Management.
2. The secret exists in this stack. Secrets are scoped to the Grafana Cloud stack. A secret created in stack A is not visible to a workflow in stack B.
3. The decrypter is correct. Workflows can only read secrets configured to allow the grafana-workflows decrypter. If you created the secret in the global Grafana Secrets Management UI, confirm Workflows is listed as an allowed decrypter.
4. The credential is still valid. A 401 from a downstream API usually means the bearer token or password expired. Update the secret value, then re-run the workflow.

For details, refer to Manage secrets.

“Incident not found” or alert group resolution failed

The engine resolves incident and alert group resources before any step runs. If resolution fails, the workflow halts before the first step.

Check, in order:

1. The ID exists in this stack. The auto-loaded incident or alert group is the one referenced in the triggering event. If the ID was deleted, resolution fails. Re-run with a fresh event.
2. The URI uses the right format. Use incident://id/<incidentID> and alertgroup://id/<alertGroupID>. The id/ segment is required.
3. The CEL expression in the URI evaluates correctly. A URI like incident://id/${inputs.data.incidentID} produces an empty ID if inputs.data.incidentID is missing or evaluates to an empty string.

For details, refer to Load incident data and Load alert group data.

A step times out

The whole workflow has a 5 minute execution timeout. A step that exceeds this budget causes the workflow to fail with a timeout.

Common causes:

• A wait step with too many seconds. Reduce the wait or split into multiple workflows.
• An HTTP call to a slow endpoint. Test the endpoint independently and reduce the response payload if possible.
• A workflow with too many steps that each take seconds to complete.

If you need a long-running automation, consider triggering subsequent workflows from the first workflow’s outputs rather than chaining everything in a single run.

Where to find the error message

Every failed run records the error inline. To read the full message:

1. Click the arrow next to Run to open Latest runs.
2. Select a failed run.
3. Find the step with a red status icon, click its kebab menu, and select View details.
4. The dialog shows the step’s error value alongside its inputs.

For details, refer to Observe workflow runs.

Next steps

• Observe workflow runs
• Sample event payloads
• Configure triggers
• Use CEL expressions
• Manage secrets