Documentation Grafana Cloud Platform Workflows Troubleshoot

Troubleshoot workflows

Use this page to diagnose the most common Workflows failures. Each section names a symptom, explains why it happens, and suggests where to look in the editor.

Step failure model

Workflows stop on the first error. When any step returns an error, the workflow run is marked Failed and no later steps run. Branches inside a switch step inherit the same behavior: if a step in the branch that runs fails, the run stops.

Workflows doesn’t retry failed steps, and you can’t configure retries in the workflow definition. To recover from a temporary failure, run the workflow again with Run. Workflows doesn’t store the triggering event with a run, so start from an example event or enter the event data. For details, refer to Trigger workflows manually.

Each step has a timeout. For most steps, it’s 5 minutes. A step that runs longer fails with a message such as step "notify-team" (slack.message.post) timed out after 5m0s, and the run fails with it. wait steps are exempt, because waiting is their purpose, and they can wait up to 7 days.

There’s no limit on how long a run can take. A run with several wait steps can stay active for days and still complete normally.

My workflow didn’t run

If your workflow doesn’t run, the event usually didn’t match the workflow’s trigger.

Check, in order:

1. The workflow is enabled. In the editor header, confirm the Enable toggle is on, and that you saved the workflow after you turned it on. A disabled workflow doesn’t run on events or schedules, and you can’t start it with Run.
2. At least one matching rule exists. In the editor’s Triggers section or the YAML startWhen.matchingRules, confirm there’s at least one eventNameRegex. A workflow with no matching rules and no schedules can’t run.
3. The regex matches the event name without the stack ID. Matching rules compare against {app}.{resource}.{event}, not {app}.{stack-id}.{resource}.{event}. A pattern like grafana_irm_app\.42\.incident\.updated never matches, because the event name that Workflows compares doesn’t include the stack ID 42. For details, refer to Configure triggers.
4. The event name uses the correct app and resource tokens. Use grafana_irm_app (with underscores), not grafana-irm-app. Use alertgroup (singular), not alert_groups. Refer to Sample event payloads for the complete list.
5. The incident isn’t private. Grafana IRM doesn’t send workflow events for private incidents, so a workflow with an incident trigger doesn’t run for them.
6. The workflow is in the same stack as the event. Workflows run only for events from their own Grafana Cloud stack.

If the workflow still doesn’t run, trigger it manually with Run to confirm the steps work. If the manual run succeeds, review the matching rule. A manual run doesn’t load the incident or alert group. If your workflow reads one, refer to A manual run fails with “no such key: incident”.

My workflow won’t save

If saving fails with the following message, the workflow has no folder:

Copy

annotation "grafana.app/folder" is required and must be non-empty

Every workflow belongs to a folder. When you save a workflow that has no folder yet, the editor opens a Select a folder dialog box. Choose a folder and save again. For details, refer to Place a workflow in a folder.

If saving fails with a permission error, check that you have edit access to the workflow’s folder. For details, refer to Manage access.

My CEL expression fails or evaluates to empty

If an expression references a field that isn’t in the data, the step fails with no such key: <KEY> and the run stops. If the field exists but its value is null, the expression evaluates to an empty string. In both cases, the expression’s path usually doesn’t match the data the step receives.

Check, in order:

1. Compare the path with the event shape. Workflows doesn’t store the triggering event with a run, and Inputs in View details shows the step’s own inputs, not the event. Compare your expression with the sample for your event type in Sample event payloads.
2. Use the right root object. Use inputs.data.* for fields from the triggering event, steps.<STEP_ID>.outputs.<FIELD> for outputs from earlier steps, and resources.incident.event.* or resources.alertgroup.event.* for auto-loaded resources.
3. Quote hyphenated step IDs with brackets. Use steps['my-step'].outputs.result instead of steps.my-step.outputs.result.
4. Escape literal ${. Use $${ to emit a literal ${ in templated strings.

For details, refer to Use CEL expressions.

A manual run fails with “no such key: incident”

A workflow that reads resources.incident or resources.alertgroup can work on real events but fail when you click Run, with no such key: incident or no such key: alertgroup. Workflows loads the incident or alert group automatically only when a Grafana IRM event triggers the run.

To test the workflow with Run, add the incident or alert group as a resource. For details, refer to Test with incident data and Test with alert group data.

My secret returned 401 or “secret not found”

If a secret is missing or Workflows can’t read it, the run fails before any steps run, so no step shows an error. The Run failed alert shows the following message:

Copy

secret verification failed: secret "<KEY>" (secret://<NAME>): <REASON>

Check, in order:

1. The secret reference uses the right URI. Confirm the URI in your workflow’s resources.secrets map matches a secret you created in the Workflows app’s Secrets page or in Grafana Secrets Management.
2. The secret exists in this stack. Secrets are scoped to the Grafana Cloud stack. A secret created in stack A isn’t visible to a workflow in stack B.
3. The decrypter is correct. Workflows can read only secrets that allow the grafana-workflows decrypter. If you created the secret in Grafana Secrets Management, confirm that grafana-workflows is an allowed decrypter.
4. The credential is still valid. An http.call step doesn’t fail when the API returns a 401. The step finishes and records the status code in responseCode, so check it in View details or in a later step with steps['<STEP_ID>'].outputs.responseCode. A 401 usually means the token or password expired. Update the secret value, then run the workflow again.

For details, refer to Manage secrets.

Workflows can’t load an incident or alert group

Workflows loads incident and alert group resources before any steps run. If it can’t load one, the run fails before the first step, and the Run failed alert shows a message that starts with failed to resolve incident resources or failed to resolve alert group resources.

Check, in order:

1. The ID exists in this stack. The auto-loaded incident or alert group is the one referenced in the triggering event. If that incident or alert group was deleted, Workflows can’t load it.
2. The URI uses the right format. Use incident://id/<INCIDENT_ID> and alertgroup://id/<ALERT_GROUP_ID>. The id/ segment is required.
3. The CEL expression in the URI evaluates correctly. A URI like incident://id/${inputs.data.incidentID} fails if inputs.data.incidentID is missing or empty.

For details, refer to Load incident data and Load alert group data.

My Ask Assistant step fails

An Ask Assistant step runs with the Grafana Cloud permissions of a specific person. For a workflow triggered on a schedule or by an event, that’s whoever last saved the workflow. For a workflow triggered manually, that’s whoever clicked Run. Workflows never falls back to a shared service identity: if it can’t identify the right person for a run, the step fails instead.

Check, in order:

1. Confirm the right person has access. Identify who last saved the workflow (or who ran it manually) and confirm that person’s Grafana Cloud permissions cover what the prompt asks the assistant to do.
2. Re-save the workflow as a user with the required access. If the workflow was last saved by someone without the needed permissions, have a user who has them open the workflow and save it again.
3. Try a manual run instead. A manual run uses your own permissions rather than those of the person who last saved the workflow. If a manual run succeeds where a scheduled or event-triggered run fails, the person who last saved the workflow doesn’t have the access the step needs.
4. Contact support if every Ask Assistant step fails. If every Ask Assistant step fails the same way, no matter who last saved or ran the workflow, Workflows might not be able to identify anyone in your stack. Contact Grafana Support.

For details, refer to Manage access.

A step times out

Each step has a timeout. For most steps, it’s 5 minutes. When a step runs longer, it fails with a message such as step "notify-team" (slack.message.post) timed out after 5m0s and no later steps run. You can’t change the timeout in the workflow definition.

Common causes:

• An HTTP call to a slow endpoint. An http.call request times out after 30 seconds, with an error that includes Client.Timeout exceeded. Test the endpoint on its own, and reduce the response size if you can.
• A code step that runs longer than its 30-second limit.
• A downstream service that accepted the request but never responded. Check whether the action happened before you run the workflow again.

wait steps are exempt from the step timeout, so a long wait isn’t the cause. If the run itself takes longer than you expect, look for a wait step with a large seconds value.

Where to find the error message

Every failed run records the error that stopped it. To read the full message:

1. Click the arrow next to Run to open Latest runs.
2. Select a failed run to expand it.
3. Read the Run failed alert at the top of the run. The alert also shows failures that happen before any steps run, such as a missing secret or an incident that Workflows can’t load.
4. For more detail on a failed step, find the step with a red status icon, click Step options (⋮), and select View details. The dialog box shows the step’s Error, Inputs, and Outputs.

For details, refer to Observe workflow runs.

Next steps

• Observe workflow runs
• Sample event payloads
• Configure triggers
• Use CEL expressions
• Manage secrets