This is documentation for the next version of Grafana documentation. For the latest stable release, go to the latest version.

Grafana Cloud Enterprise Open source

Use gcx to manage alerting resources

gcx is a command-line interface (CLI) that you and coding agents can use to manage Grafana resources. It provides two ways to manage alerting resources:

  • gcx alert commands to inspect and manage alerting resources, such as contact points, notification policies, and templates. Use them, from the CLI or with a coding agent, to check the state of your alert rules, review alert instances and history, make one-off changes, and manage resources that gcx resources doesn’t support, such as contact points.

  • gcx resources commands to export and provision alerting resources as manifest files. Use them to keep alert rules in version control, review changes before you apply them, and apply the same alert rules to other Grafana instances. Provisioned alert rules and recording rules cannot be edited in the Grafana UI.

Before you begin

  1. Install gcx.

  2. Authenticate gcx with the preferred method for your Grafana instance.

  3. To check which configuration gcx uses, run:

    Bash
    gcx config view

Manage resources with gcx alert

Use gcx alert to inspect and manage alerting resources. Each resource type has its own commands.

ResourcePurposeOperations
contact-pointsManage contact pointslist, get, create, update, delete, export
groupsInspect Grafana-managed alert rule groups (read-only)list, get, status
instancesList alert instances (read-only)list
mute-timingsManage mute timingslist, get, create, update, delete, export
notification-historyInspect delivered alert notifications (read-only)list, list-alerts
notification-policiesManage the notification policy treeget, set, reset, export
routing-treesManage notification routing trees, including the default treelist, get, create, update, delete
rulerManage data source-managed (Mimir and Loki) rulesgroups, namespaces
rulesInspect Grafana-managed alert rule state and health (read-only)list, get
state-historyQuery alert rule state transitions (read-only)list
template-groupsManage native notification template groupslist, get, create, update, delete
templatesManage notification templateslist, get, upsert, delete
time-intervalsManage notification time intervalslist, get, create, update, delete

Note

The gcx alert commands are in active development.

Run gcx alert --help or gcx alert <RESOURCE> --help to see what your version supports.

Caution

Contact points can contain credentials.

Grafana exports secure fields as [REDACTED], so exported files don’t contain secrets. Replace these placeholders with the required values before provisioning.

Manage resources using CRUD operations

Use the operations available for each resource type to create, read, update, or delete resources. The following examples use contact points:

  • Create a contact point from a file. For example, save the following definition as contact-point.yaml:

    YAML
    name: team-webhook
    type: webhook
    settings:
      url: https://example.com/alerts

    Then create the contact point:

    Bash
    gcx alert contact-points create --filename contact-point.yaml
  • List contact points to verify that the new one exists:

    Bash
    gcx alert contact-points list

    This command returns the contact points with their UID, name, type, and provenance state.

    ┌──────────────────┬─────────────────┬──────────┬─────────────┐
    │ UID              │ NAME            │ TYPE     │ PROVENANCE  │
    ├────-─────────────┼──────-──────────┼──────────┼─────────────┤
    │ dg40y30b347wgb   │ team-webhook    | webhook  │             │
    └──────-───────────┴─────────────────┴──────────┴─────────────┘
  • Get an existing contact point and save it to a file for editing. Use -o (--output) to choose the output format, json or yaml:

    Bash
    gcx alert contact-points get <UID> -o yaml > contact-point.yaml
  • Update a contact point. Edit the file and run update with the contact point UID:

    Bash
    gcx alert contact-points update <UID> --filename contact-point.yaml
  • Delete a contact point by its UID:

    Bash
    gcx alert contact-points delete <UID>

    gcx asks for confirmation before it deletes the contact point. To skip the prompt, add --force.

Some operations replace or update existing data:

Export resources for provisioning

Use export to save existing resources in the Grafana provisioning format:

Bash
gcx alert contact-points export --format yaml > contact-points.yaml

Use the yaml format with file provisioning or hcl with Terraform.

Disable updates in the Grafana UI

By default, resources written by gcx alert are editable in the Grafana UI.

To prevent UI updates so that a resource can only be changed through provisioning, set --allow-ui-updates=false:

Bash
gcx alert contact-points create --filename contact-point.yaml --allow-ui-updates=false

The allow-ui-updates flag is available on the following commands:

  • gcx alert contact-points create and update
  • gcx alert mute-timings create and update
  • gcx alert templates upsert
  • gcx alert notification-policies set

Manage resources with gcx resources

Use gcx resources to export and provision alerting resources as manifest files that you can store in version control.

Use pull to export existing alert rules to a local directory:

Bash
gcx resources pull alertrules --path ./config/
output
✔ 3 resources pulled, 0 errors

By default, gcx resources pull writes JSON and names each file after metadata.name.

Use push to provision the alert rules saved into the directory.

Bash
gcx resources push alertrules --path ./config/
output
✔ 3 resources pushed, 0 errors

To push one resource, add its metadata.name to the selector:

Bash
gcx resources push alertrules/<UID> --path ./config/
output
✔ 1 resources pushed, 0 errors

Note

Pushing an existing resource replaces it.

Provisioned alert rules and recording rules are marked as Provisioned in the Grafana UI and cannot be edited there. This prevents UI changes that would be overwritten when the resource is provisioned again.

Supported alerting resources

gcx resources supports a subset of alerting resource types.

The supported types depend on the API provided by your Grafana version. Resources that include secrets, such as contact points, are currently excluded.

To list the resource types your instance serves, run:

Bash
gcx resources list-types --no-schema -o text | grep alerting | grep -w v1beta1
output
notifications.alerting.grafana.app   v1beta1   routingtrees
notifications.alerting.grafana.app   v1beta1   templategroups
notifications.alerting.grafana.app   v1beta1   timeintervals
rules.alerting.grafana.app           v1beta1   alertrules
rules.alerting.grafana.app           v1beta1   recordingrules
rules.alerting.grafana.app           v1beta1   rulesequences

Provision alert rules into a new Grafana instance

Before pushing alert rules to a new Grafana instance, check the following requirements.

  1. Include the folders.

    Alert rules belong to a folder, and that folder must exist in the target instance. If a folder is missing, the push fails with the following error:

    output
    403 Forbidden: alertrules.rules.alerting.grafana.app "<UID>" is forbidden: folder does not exist

    You can pull folders and alert rules into the same directory and push them together. gcx pushes folders before other resources:

    Bash
    gcx resources pull folders alertrules --path ./config/
    gcx --context <NEW_INSTANCE> resources push folders alertrules --path ./config/
  2. Remove the group labels.

    Pulled rules might include the grafana.com/group and grafana.com/group-index labels. The target instance rejects rules with these labels when creating them:

    output
    403 Forbidden: alertrules.rules.alerting.grafana.app "<UID>" is forbidden: cannot set group when creating a new rule

    If the push fails with this error, remove the grafana.com/group and grafana.com/group-index labels from the alert rule files and push again.