Documentation Grafana Cloud Platform Workflows Reference Blueprints Run GitHub Actions remediation

Run GitHub Actions remediation

Run a GitHub Actions remediation workflow, such as a service restart, when an alert group labeled for automatic remediation fires.

This Grafana Workflows blueprint is in the Integrations category of the Blueprints library.

Before you begin

To use this blueprint, you need:

• Grafana IRM set up on your stack.
• Slack connected to Grafana IRM. For details, refer to Configure Slack for Grafana IRM.
• A workflow secret named github-actions-token.

To create a secret, refer to Manage secrets.

Steps

Steps run in order from first to last. A step inside a switch branch runs only when Workflows chooses that branch.

For the inputs and outputs of each step type, refer to Step and trigger types reference.

Before you turn on the workflow, customize any steps that need input from you, such as a channel ID or a schedule name. Each step’s description says what to set.

Workflow definition

Save this definition as a YAML file to create the workflow with gcx, the Workflows API, or Terraform, or give it to a coding agent as an example. Replace FOLDER_UID with the UID of the folder for the workflow. Blueprints start disabled, so finish customizing the steps before you set enabled to true. For details, refer to Manage workflows as code.

YAML Copy

apiVersion: workflows.ext.grafana.app/v1alpha1
kind: Definition
metadata:
  name: run-github-actions-remediation
  annotations:
    grafana.app/folder: <FOLDER_UID>
spec:
  name: Run GitHub Actions remediation
  enabled: false
  runOnceFor: 'remediate:${inputs.data.alertGroupID}'
  startWhen:
    matchingRules:
      - eventNameRegex: 'grafana_irm_app\.alertgroup\.updated'
        filter: "inputs.data.eventKind == 'created' && resources.alertgroup.event.labels.exists(l, l.key.name == 'auto_remediate' && l.value.name == 'true')"
    schedules: []
  resources:
    secrets:
      githubActionsToken: 'secret://github-actions-token'
  steps:
    - id: dispatch
      type: http.call
      name: Start the GitHub workflow
      description: >-
        Starts your remediation workflow. Replace your-org/your-repo with the repository and
        your-workflow.yml with the workflow file to run. Also create a workflow secret named
        github-actions-token with a token that has Actions write access to that repository.
        Change ref if the workflow lives on a branch other than main. The service input comes
        from the alert group's service label, so validate it in your workflow.
      inputs:
        method: 'POST'
        url: 'https://api.github.com/repos/your-org/your-repo/actions/workflows/your-workflow.yml/dispatches'
        headers: '["Accept: application/vnd.github+json", "X-GitHub-Api-Version: 2022-11-28"]'
        body: |
          {
            "ref": "main",
            "inputs": {
              "service": ${json_stringify(resources.alertgroup.event.labels.exists(l, l.key.name == 'service') ? resources.alertgroup.event.labels.filter(l, l.key.name == 'service')[0].value.name : '')},
              "alert_group_id": "${inputs.data.alertGroupID}"
            }
          }
        authMethod: 'bearer'
        authSecret: '${resources.secret.githubActionsToken}'
    - id: check-dispatched
      type: filter
      name: Fail if GitHub rejected the request
      description: Fails the run with an error unless GitHub accepted the dispatch. No changes needed.
      inputs:
        condition: 'steps.dispatch.outputs.responseCode == 204'
        failOnExit: 'true'
    - id: announce
      type: slack.message.post
      name: Tell the team
      description: Lets the team know an automatic remediation started. Set channelID.
      inputs:
        channelID: ''
        messageText: ':robot_face: Started automatic remediation for <${resources.alertgroup.event.permalinks.web}|${resources.alertgroup.event.title}>.'

How it works

This section explains when the workflow runs and how its steps work together.

When it runs

This blueprint starts on the grafana_irm_app.alertgroup.updated event.

Its trigger filter starts a run only when this CEL expression is true:

text Copy

inputs.data.eventKind == 'created' && resources.alertgroup.event.labels.exists(l, l.key.name == 'auto_remediate' && l.value.name == 'true')

It runs at most once for each run-once key. This blueprint’s key is remediate:${inputs.data.alertGroupID}, so a later event with the same key doesn’t start another run. For details, refer to Deduplicate workflow runs.

How the steps fit together

These notes cover how the steps connect and which Workflows behaviors the blueprint relies on.

• Triggers on grafana_irm_app.alertgroup.updated with eventKind created, and only for alert groups labeled auto_remediate=true.
• The label only switches the remediation on.
• The GitHub Actions workflow file is fixed in the step’s URL, so whoever can set labels can’t choose which workflow runs with the token.
• runOnceFor keys on the alert group ID, so each alert group dispatches at most once.
• The service input is read from the alert group’s service label with CEL filter() over resources.alertgroup.event.labels.
• Label values are free text, so the GitHub workflow should validate it.
• http.call POSTs to the GitHub workflow_dispatch API with a fine-grained token stored as a workflow secret and referenced only in authSecret.
• Extra GitHub headers go in headers as a JSON array of ‘Key: Value’ strings.
• GitHub returns 204 on success, so a filter with failOnExit fails the run on any other status.
• Only label alerts whose remediation is safe to run unattended.
• Customize the repository and workflow file in the URL, the branch ref, the workflow inputs, and the Slack channel.
• Copy the blueprint for each remediation workflow you want to run.

Next steps

• Blueprints
• Start from a blueprint
• Manage workflows as code
• Step and trigger types reference