User guides on Grafana Labs

Dashboards as code

Mon, 12 Oct 2026 15:20:54 +0000

Manage dashboards as code with `gcx`

With this workflow, you can define and manage dashboards as code, saving them to a version control system like Git. This is useful for teams that want to maintain a history of changes, collaborate on dashboard design, and ensure consistency across environments.

Before you begin

In order to use the gcx dev serve functionality, you need to enable a feature toggle in your configuration file:

[feature_toggles]
kubernetesDashboards = true

Refer to Configure Grafana to learn more about enabling feature toggles.

Although the local preview proxy supports service account token, Basic, and mTLS contexts, use a service account token instead of OAuth for gcx dev serve. OAuth is rejected before startup until refresh-token rotation can be persisted safely by the long-running server.

Workflow

To manage dashboards as code with gcx follow these steps:

Use a dashboard generation script (for example, with the Foundation SDK). You can find an example implementation in the Grafana as code hands-on lab repository.

Serve and preview the output of the dashboard generator locally:

gcx config use-context YOUR_CONTEXT  # for example "dev"
gcx dev serve --script 'go run scripts/generate-dashboard.go' --watch './scripts'

When the output looks correct, generate dashboard manifest files:

go run scripts/generate-dashboard.go --generate-resource-manifests --output './resources'

Push the generated resources to your Grafana instance:

gcx config use-context YOUR_CONTEXT  # for example "dev"
gcx resources push -p ./resources/

Explore and modify resources

Wed, 04 Nov 2026 15:24:35 +0000

Explore and modify resources with `gcx`

gcx allows you to interact with Grafana resources directly from your terminal. You can browse, inspect, update, and delete resources without using the Grafana UI.

Use this approach if you’re an advanced user and want to manage resources more efficiently, or to integrate Grafana operations into automated workflows.

Search resources by relevance

Use the experimental gcx resources search command to find ranked matches from a phrase:

gcx resources search "kubernetes memory problems"

Search returns top matches, without pagination or an exhaustive resource inventory. Scores rank matches in the response.

Prerequisites

You need to enable the Grafana global hybrid search endpoint to use gcx resources search.

On supported Grafana Cloud stacks, gcx resources search can combine text and semantic matching.
On self-managed deployments with the endpoint, only use text matching. Semantic matching isn’t available on-premises.

Your deployment must have default resource types, or you must select types with --resource.

Search for specific resources

Select resource types and exact folder UIDs for a narrower search:

gcx resources search "memory" \
  --resource dashboards.dashboard.grafana.app \
  --resource folders.folder.grafana.app \
  --folder team-platform \
  --limit 10
gcx resources search "memory" --folder general

Repeat --resource to search your selected types instead of the server defaults.

Repeat --folder to match any listed folder UID. Folders match exactly, without descendants; general or --folder '' selects the root folder. The default limit is 50, and the maximum is 200.

Use -o wide to see matching text, or -o json or -o yaml for the full response:

gcx resources search "memory" -o wide
gcx resources search "memory" -o json

Find and delete dashboards using incorrect data sources

To identify and remove production dashboards relying on non-production data sources, use the command below to list dashboard UIDs along with the data source UIDs used in their panels:

gcx resources get dashboards --context prod | jq '.items | map({ uid: .metadata.name, datasources: .spec.panels | map(.datasource.uid)  })'
[
   {
      "uid": "important-production-dashboard",
      "datasources": [
         "mimir-prod"
      ]
   },
   {
      "uid": "test-dashboard-from-dev",
      "datasources": [
         "mimir-dev"
      ]
   },
   {
      "uid": "test-dashboard-from-stg",
      "datasources": [
         "mimir-stg"
      ]
   }
]

Next, identify the dashboards that are using unexpected data sources, and delete them:

gcx resources delete dashboards/test-dashboard-from-stg,test-dashboard-from-dev
✔ 2 resources deleted, 0 errors

Edit remote resources

You can edit resources directly from the default editor, without having to pull them first. To do so, run:

gcx resources edit dashboard/edit-me-please

This command opens the default editor set in the EDITOR environment variable, or falls back to ‘vi’ for Linux and ’notepad’ for Windows.

After the editor process terminates, the resource is updated in the Grafana instance targeted by the current context. Edition is cancelled if no changes are written to the file, or if the file after edition is empty.

Manage resources

Wed, 04 Nov 2026 15:24:35 +0000

Manage resources with `gcx`

Migrate resources between environments

Grafana resources can be migrated from one environment to another, for example from a development to production environment.

You need to configure one context per environment. In this example scenario, use dev for the development environment and prod for production.

Make changes to dashboards and other resources using the Grafana UI in your development instance.
Pull those resources from the development environment to your local machine:
shell
```
gcx resources pull --context dev # Add `-o yaml` export resources as YAML 
```
Push the resources to production:
shell
```
gcx resources push --context prod
```

Note
Resources are pulled and pushed from the ./resources directory by default. Configure this path with the --path/-p flags.

Name pulled files by title

By default, gcx resources pull names each file using the resource’s metadata.name (usually its UID). Use --filename-by=title for readable filenames:

gcx resources pull dashboards --filename-by=title --path ./dashboard-backup

For example, a dashboard titled Ventas Overview becomes ventas-overview.json inside its usual resource-type directory. Resources without a usable title fall back to metadata.name. The resource contents retain their original identity, so push, validate, and serve use these files normally.

Title mode resolves filename conflicts, preserving files that belong to other resources. See the file naming rules for the full algorithm.

PULL doesn’t remove old files. Title-mode pull warns on stderr when it finds another file for the same resource at a different path, such as after a rename or switching naming modes. Reading resources also warns when skipping duplicate identities. Pull into a clean directory or remove stale files (for example, with git rm) before pushing.

Back up and restore resources

Follow this workflow to back up all Grafana resources from one instance and later restore them. This can be useful to replicate a configuration or perform disaster recovery.

Ensure the current context points to the Grafana instance to backup/restore:
shell
```
gcx config use-context YOUR_CONTEXT  # for example "prod"
```

Pull all resources from your target environment:

gcx resources pull --path ./backup-prod # Add `-o yaml` to export resources as YAML

Save the exported resources to version control or cloud storage.
Push the resources to restore them:
shell
```
gcx resources push --path ./backup-prod
```

Incident descriptions

Incident descriptions remain visible in GET and LIST results.

PULL omits descriptions from saved manifests. You cannot restore descriptions if you backed up an incident with gcx resources pull.

Old backup files can contain spec.description. Remove that field before you push the file to create an incident or to change an incident with a different description. You can still update other fields.

Lint resources

Mon, 12 Oct 2026 15:20:54 +0000

Lint resources with `gcx`

Use the Grafana CLI linter to verify that resources, such as dashboards and alert rules, comply with good practices and environment-specific policies.

Use the linter

To lint resources run:

gcx dev lint run ./resources

The following applies:

Directories are recursively explored.
All built-in rules are enabled by default.

Configure linter rules

For a finer control, you can configure the linter rules:

# Disable all rules for a resource type:
gcx dev lint run --disable-resource dashboard ./resources

# Disable all rules in a category:
gcx dev lint run --disable-category idiomatic ./resources

# Disable specific rules:
gcx dev lint run --disable uneditable-dashboard --disable panel-title-description ./resources

# Enable rules for specific resource types:
gcx dev lint run --disable-all --enable-resource dashboard ./resources

# Enable only some categories:
gcx dev lint run --disable-all --enable-category idiomatic ./resources

# Enable only specific rules:
gcx dev lint run --disable-all --enable uneditable-dashboard ./resources

You can modify the severity level of a rule by updating the custom.severity annotation. Valid values are warning and error.

Define custom linting rules

Custom and built-in rules are defined in Rego, the policy language used by Open Policy Agent (OPA). This ensures that resources comply with policies specific to your environment.

You can scaffold new custom rules with gcx:

# Creates a new "dashboard" linter rule in the current directory:
gcx dev lint new dashboard custom-rule

This generates a file with the bootstrapped rule:

# METADATA
# description: Briefly describe the rule here.
# custom:
#  severity: warning
package custom.gcx.rules.dashboard.idiomatic["custom-rule"]

import data.gcx.result
import data.gcx.utils

# Dashboard v1
report contains violation if {
	utils.resource_is_dashboard_v1(input)

	input.spec.timezone != "utc"

	violation := result.fail(rego.metadata.chain(), sprintf("timezone is '%s', expected 'utc'", input.spec.timezone))
}

# Dashboard v2
report contains violation if {
	utils.resource_is_dashboard_v2(input)

	input.spec.timeSettings.timezone != "utc"

	violation := result.fail(rego.metadata.chain(), sprintf("timezone is '%s', expected 'utc'", input.spec.timeSettings.timezone))
}

Refer to the existing Built-in rules for inspiration to write custom ones.

Rules for other resources

You can create rules for other resource types than dashboards, or in categories other than idiomatic:

# Creates a new "alertrule" linter rule, categorized under "bug":
gcx dev lint new alertrule custom-rule -c bug

Diagnose missing telemetry

Mon, 12 Oct 2026 15:20:54 +0000

Diagnose missing telemetry with `gcx`

If you’re investigating an empty dashboard or missing application telemetry, use this guide to learn how to use gcx to gather evidence before proposing a repair. You don’t need a new diagnostic skill or Grafana Cloud.

You can investigate directly or use an agent with gcx existing skills. Start with evidence, ask before repairs, and verify the original symptom afterward. If you use an agent, its harness and model access are separate from gcx.

Before you begin

Before you begin, make sure to:

Follow Install gcx if gcx version is unavailable.
Follow Install Agent Skills for your harness. Start a session that can discover the installed skills. The existing debug-with-grafana skill handles investigation; setup-gcx helps with setup.
Use a reachable Grafana instance supported by your gcx version, with access to the relevant datasources. A static instrumentation check alone does not provide a telemetry destination.
If you have no backend, start docker-otel-lgtm for a local development sink. Setting up a sink does not authorize changing an existing application’s exporter. A local proof does not establish that your production destination works.
Know which application or fixture you may inspect. Do not enumerate unrelated containers, environments, or services to compensate for missing context.

The command examples use the current gcx command surface. Check gcx version and the relevant command’s --help when following them with another release.

Connect without changing your usual context

If you have already configured the destination, use explicit --config and --context arguments. Don’t switch the current context just for this investigation. Otherwise, create a separate private configuration outside your checkout.

The examples use GCX_CONFIG, the actual gcx environment override. If it’s pointing at another configuration, unset it first:

GCX_DIAGNOSTICS_DIR=$(mktemp -d)
export GCX_CONFIG="$GCX_DIAGNOSTICS_DIR/config.yaml"
(umask 077; printf '{}\n' > "$GCX_CONFIG")

Keep the path until cleanup. In another terminal, set the variable to the same path; don’t create a second empty configuration by repeating the block.

Log in

For an existing Grafana destination, use interactive login with a new context in that file, replacing the example URL:

gcx login diagnostics --config "$GCX_CONFIG" \
  --server https://your-grafana.example

For local LGTM, use that repository’s local connection instructions instead of this login step. Refer to Configure gcx for supported authentication methods. Supply credentials locally through the supported authentication flow, not in chat, copied transcripts, or committed files. Prefer read-only access when available as approval instructions don’t restrict API permissions.

Check the overrides

Environment overrides can still affect the selected configuration. Review which overrides you intentionally set before running queries and don’t dump your whole environment or configuration into a transcript. Don’t disable TLS verification to make a connection error disappear.

Use your selected context name below, where diagnostics is the login example:

gcx config check --config "$GCX_CONFIG" --context diagnostics
gcx datasources list --config "$GCX_CONFIG" --context diagnostics

Verify the connection and authentication

Stop and resolve any connection or authentication errors before interpreting query results.

For example, against a Prometheus datasource:

gcx metrics query 'vector(1)' --datasource 'PROMETHEUS_DATASOURCE_UID' \
  --error-on-empty \
  --config "$GCX_CONFIG" --context diagnostics

Keep in mind the following:

Choose the datasource UID returned by discovery, rather than assuming a default.
A successful configuration check doesn’t prove application ingestion.
A returned value proves that query path works, not that the application emitted metrics.

With --error-on-empty, an empty response exits unsuccessfully; without it, a successful empty query, an invalid query, and a failed connection are different observations. The flag is supported on the Prometheus query, Loki logs/metrics, Tempo search/metrics, and Pyroscope query commands, but it’s not a generic flag for unrelated datasource query commands.

Use an agent for diagnostics

Start the agent in the relevant application or test checkout, and provide it with the symptoms and boundaries for diagnostics:

The original dashboard URL, failed test assertion, or checker finding.
The explicit gcx config path, context, and intended destination (no secrets).
The application/service identity and one representative operation, if known.
The UTC time window, observed query, and request/trace identifiers, if available.
The relevant Compose project, deployment, or Collector configuration you own.
Which resources may be inspected and which actions require approval.

Make sure the agent doesn’t invent answers and to identify what information or access is missing. For example, use this prompt, filling in what you know:

Use the existing gcx skills to investigate `<symptom>`. Query only `<config path and context>` and inspect only `<application/fixture scope>`. The expected operation is `<operation>` in `<time window>`. Diagnose first and distinguish observed evidence from hypotheses. State what you could not observe; do not infer data loss from missing access. Ask before changing application, Collector, dashboard, or test configuration, restarting resources, or enabling payload logging. Do not weaken an assertion to make a test pass. Propose the smallest justified repair, and stop for approval.

An empty dashboard can result from no data being created, export failure, filtering, an incorrect query, or a time/identity mismatch. Static configuration and absence of results are leads, not proof of a particular failure boundary.

Always ask for the evidence supporting each conclusion and the next discriminating check.

Verify a repair and restore the environment

After approving a specific change:

Exercise a fresh representative operation and record its time and available identifiers. Old telemetry must not satisfy the recovery check.
Query the relevant datasource, then recheck the original dashboard or rerun the original test uncached. A successful backend query alone does not prove the panel or assertion is correct.
Review the actual diff and runtime changes. Functional recovery doesn’t prove that unrelated dashboard or configuration fields were preserved.
Restore temporary diagnostic configuration and logging. Remove temporary backups and stop only resources created for this investigation, after confirming they are no longer needed. Never clean up pre-existing resources by assumption.
Record the confirmed cause, remaining unknowns, approved changes, and result.

Debug-exporter payloads and logs may contain sensitive data. Enable them only with permission, for a bounded investigation, and never publish raw output. Avoid full environment dumps or HTTP payload logging that can expose credentials.

If you created the private directory above, inspect it locally and remove only that directory when finished. Remove any diagnostic credentials stored by login using your normal credential-management process; deleting a config file doesn’t revoke a server-side token. Explicit configuration or context arguments leave your usual current context unchanged. Don’t delete an existing configuration used for the investigation, or revoke a credential shared with another workflow.

Manage Machine Learning resources

Mon, 02 Nov 2026 08:26:02 +0000

Manage Machine Learning resources

Use gcx ml to manage Grafana Cloud Machine Learning resources.

Prerequisites

The following applies:

Your Grafana stack must have Machine Learning enabled.
You don’t need a separate provider token. The commands use the active Grafana credential through the Machine Learning plugin proxy.

Inspect resources

Forecasts are jobs in the Machine Learning API.

Use the server IDs from the list output with get, update, and delete.

gcx ml jobs list
gcx ml holidays list
gcx ml outlier-detectors list
gcx ml jobs get <job-id> -o yaml

Use --json or --jq to select output fields.

Configure the number of records in the output

Lists fetch all records from the collection endpoint and show up to 50 records by default:

Use --limit 0 to show all records.
Use --limit 10 to show fewer records. A truncation notice appears on stderr when records are omitted.

Apply a resource

Use a resource example to create a manifest. Set the datasource UID and the query for your target stack:

gcx resources list-examples jobs.ml.ext.grafana.app --jq '.["ml.ext.grafana.app"].v1alpha1[0].example' -o json > job.json
gcx ml jobs create -f job.json --dry-run
gcx ml jobs create -f job.json
gcx ml jobs update <job-id> -f job.json

Creating or updating a forecast job can start model training.

--dry-run checks the input without a server mutation. The result reports that server validation is not available.

Delete a resource

Delete commands require confirmation or --force.

Export and push manifests

Jobs, holidays, and outlier detectors use the standard resource pipeline. The provider removes server IDs and connection fields from each spec.

Target stack

Each write uses the target stack URL. The server ID from the source stack is stored in metadata.name. If that ID doesn’t exist on the target, the pipeline matches the normalized value of spec.name.

Each write resolves the numeric datasource ID from the UID on the target stack. Writes need read access to the selected datasource.

Matching:

Converts names to lowercase slugs.
Updates one matching resource or creates a resource when there is no match.
Multiple matches are rejected.

Push cross-stack

Data source UIDs and job-to-holiday references remain in the spec. Set these references for the target stack before a cross-stack push.

After a cross-stack push, pull from the target to save its assigned IDs:

gcx resources push -p ./ml --dry-run
gcx resources push -p ./ml
gcx resources pull jobs.ml.ext.grafana.app holidays.ml.ext.grafana.app outlierdetectors.ml.ext.grafana.app -p ./ml

Manage parent alerts

Machine Learning alerts belong to a job or an outlier detector, and have no independent collection endpoint or resource adapter. Use the parent ID with the alert commands.

gcx ml jobs list-alerts <job-id>
gcx ml jobs get-alert <job-id> <alert-id> -o yaml
gcx ml jobs create-alert <job-id> -f alert.yaml
gcx ml jobs update-alert <job-id> <alert-id> -f alert.yaml
gcx ml jobs delete-alert <job-id> <alert-id> --force
gcx ml outlier-detectors list-alerts <detector-id>

The same alert operations are available under outlier-detectors.

Alert objects

Alert files contain the Machine Learning alert object, including its title.

Use get-alert to export an alert object before you edit it.

Outlier alerts

Outlier alerts require a nonzero window, such as 5m. Don’t set threshold or anomalyCondition for an outlier alert.

Manage OnCall notification rules

Mon, 02 Nov 2026 08:26:02 +0000

Manage OnCall personal notification rules

You can use gcx to manage OnCall personal notification rules

Authentication and permissions

Use gcx login to sign in as a Grafana user.

The command uses the shared Grafana authentication transport and the IRM plugin proxy, and it doesn’t need a separate OnCall API key.
Service account requests are rejected by the server.

The server requires grafana-oncall-app.user-settings:read for reads and grafana-oncall-app.user-settings:write for changes. Changes to another user’s rules also require grafana-oncall-app.user-settings:admin.

Manage OnCall notification rules

To get a list of rules for all users and priority levels, run:

gcx irm oncall personal-notification-rules list
gcx irm oncall personal-notification-rules get <rule-id>
gcx resources list-examples personalnotificationrules

Manage rules with these commands:

Create rules with create -f rule.yaml.
Update rules with update <rule-id> -f rule.yaml.
Delete rules with delete <rule-id>.

Rule fields

Do not use the public API’s type or duration field names. step, notify_by, and wait_delay use the internal IRM API values.

Rule fields such as user and important are set when the rule is created, and are ignored in updates.

Use a string of seconds for wait_delay, such as "300.0", or null.

The server assigns its own rule ID at creation. Use the returned ID for later updates. For resource pushes, pull the created rule and keep that manifest before the next push. Its metadata.name contains the server ID. The server also controls order.

The API can also return notify_device. gcx preserves this server field in exports and updates. It’s null when the rule has no selected device.

Routes

The proxy route is notification_policies/, and uses the same UserNotificationPolicy model as the public OnCall /api/v1/personal_notification_rules/ route.

The public route requires an OnCall API token in the raw Authorization header. A Grafana OAuth token is used through the plugin proxy instead.

The internal route defaults to the current user and important=false when query parameters are absent.

gcx sends ?user=&important= to disable these defaults. The backend then returns the full collection. The resource adapter uses the same client methods as the provider commands.

Select a step and notification channel

Use these internal API values for step:

Value	Step
`0`	Wait for `wait_delay` seconds
`1`	Notify through `notify_by`

A wait step returns notify_by: null. The default table shows a dash for that value and shows the wait duration in WAIT-DELAY.

Notification channels

Notification channels depend on the stack and its integrations. Read the live catalog to get the available values and display names:

gcx api /api/plugins/grafana-irm-app/resources/notification_policies/notify_by_options/

Common notify_by values are:

Value	Channel
`0`	Slack mentions
`1`	SMS
`2`	Phone call
`3`	Telegram
`5`	Mobile push
`6`	Mobile push important
`7`	Microsoft Teams
`8`	Email
`11`	Webhook

Compare profiles

Wed, 28 Oct 2026 16:28:10 +0000

Compare profiles with `gcx`

You can compare profiles with gcx:

Use gcx profiles query for one profile.
Use gcx profiles diff for symbol costs across selectors, time windows, or both. diff uses Pyroscope’s server-side comparison, not independently truncated top-function lists.

Follow these steps to compare profiles:

Configure a Grafana context with read access to Pyroscope.
Find the datasource UID with gcx datasources list --type pyroscope.
Find the profile type with gcx profiles list-profile-types -d <datasource-uid>.
Run then following commands to compare:

gcx profiles diff '{service_name="frontend",version="v1"}' '{service_name="frontend",version="v2"}' \
  -d <datasource-uid> --profile-type process_cpu:cpu:nanoseconds:cpu:nanoseconds \
  --before-from now-1h --before-to now \
  --after-from now-1h --after-to now --drilldown-link

In this example, both sides share a datasource and profile type. Repeat the same selector to compare different windows. Unsupported selectors warn and fall back without dropping scope.

Use fixed timestamps for reproducible evidence. All four bounds are required: RFC3339, Unix seconds, or relative times. They use one clock snapshot and resolve to whole seconds to match Drilldown.

Dedicated comparisons require the same exact service_name. Names requiring string escaping fall back to Explore, while other matchers are preserved separately.

For a profile UUID from exemplars, add --profile-id <uuid> --drilldown-link to gcx profiles query. A single ID opens the scoped flame graph and multiple IDs or stacktrace selectors fall back to Explore. Trace-ID-scoped queries suppress links because neither UI can preserve their scope.

Interpret costs

Table-wide output shows the top 20 absolute self-cost changes.

JSON, YAML, and agents output retain all returned symbols, warnings, selections, units, and links. The backend node budget defaults to 50,000 in every format.

Take into account the following to interpret costs:

Costs use the profile type’s raw sample unit.
Recursive inclusive costs count only the outermost occurrence per stack.
self attributes cost directly to a function.
total includes callees.
delta is after minus before, without duration or traffic normalization.
Prefer comparable, equal-length windows: traffic, replicas, and sampling can change cost without a regression.
changePercent is null for zero baselines or when either profile has no positive samples. Missing samples do not establish zero cost or improvement.
complete=false means frames were folded into other; omitted reports their costs.
Raise --max-nodes or narrow selectors to improve attribution.
Absence from a pruned graph does not establish an added or removed function.

To share evidence:

links.before and links.after open individual profiles.
links.explore opens before on the left and after on the right, with independent ranges.
links.comparison opens the dedicated diff view when representable.

The following options are available:

--share-link / --open: print or open split Explore.
--drilldown-link / --open-drilldown: print or open Profiles Drilldown. You must enable Drilldown, since URLs target its Scenes contract used in version 3.1.0. Use Explore for older apps that don’t support that contract.

Manage Assistant Watchers

Thu, 22 Oct 2026 08:20:30 +0000

Manage Assistant Watchers

Use gcx to create Watchers and keep their configuration in version control. Watcher commands are experimental and require Grafana Cloud.

Before you begin

Configure a gcx context for a Grafana Cloud stack with Assistant Watchers enabled.
Make sure your identity can read and manage Watchers and access the data sources you want to monitor.

Commands use your selected context. Add --context CONTEXT to select another context.

Create a Watcher and follow calibration

To create a Watcher, follow these steps:

Save the following manifest as watcher.yaml. Replace DATASOURCE_UID with an accessible data source UID.

apiVersion: assistant.ext.grafana.app/v1alpha1
kind: Watcher
metadata:
  name: checkout-health
spec:
  title: Checkout health
  prompt: Watch request failures and sustained latency increases.
  datasourceUids: [DATASOURCE_UID]
  interval: 15m
  sensitivity: balanced

Create the Watcher:
shell
```
gcx assistant watchers create -f watcher.yaml -o json
```
The result includes the saved resource and a separate calibration request outcome. Creation requests background calibration without automatically starting monitoring. Notification destinations and investigations are disabled unless you enable them in the manifest. Watcher calibration and runs consume Grafana Assistant tokens; refer to Assistant pricing.
Replace WATCHER with the returned resource name or ID and inspect progress:
shell
```
gcx assistant watchers status WATCHER -o json
```
An accepted request means calibration was requested, not that checks are ready. Wait while it is queued or running. If calibration fails, is cancelled, or needs input, continue in Grafana. When it completes, review the enabled checks in Grafana before starting monitoring.

If the request outcome is unavailable, calibrate the saved Watcher in Grafana. An error or not-requested outcome returns exit code 4 and retains the Watcher identity. Inspect the reported failure and status; resolve any failed configuration stage with update, then continue in Grafana. Do not repeat create: a request error can leave calibration running.

Inspect and update configuration

To change an existing Watcher, follow these steps:

Find its resource name or ID:
shell
```
gcx assistant watchers list
```
Add --archived to list archived Watchers. If a name matches several Watchers, use an ID.

Save its configuration to a file:

gcx assistant watchers get WATCHER -o yaml > watcher.yaml

Edit the file, then apply the complete manifest:
shell
```
gcx assistant watchers update WATCHER -f watcher.yaml
```

update and resources push replace the modeled configuration. Omitted optional settings receive defaults, including disabled notification destinations. Start from an exported manifest to retain your existing choices. These operations preserve calibrated checks and learned state, and do not request calibration, start monitoring, or pause it. If an edit is rejected during calibration, wait for it to finish or cancel it in Grafana.

If an apply reports a partial failure, changes may already be saved. Inspect the returned Watcher ID and failed stage, resolve the failure, and reapply with update or resources push.

Use status for calibration, checks, and runtime observations; get exports configuration only. A paused Watcher can retain an older health assessment.

Use Grafana for custom automatic-recalibration schedules and other settings gcx cannot represent. gcx refuses configuration writes that would drop those settings.

Manage configuration files

To store and reapply Watcher configuration, follow these steps:

Export the manifests:

gcx resources pull watchers -p ./watchers -o yaml

Edit the files and validate them without writing to Grafana:
shell
```
gcx resources push -p ./watchers --dry-run
```
Review your changes, then apply them:
shell
```
gcx resources push -p ./watchers
```

An unchanged apply sends no mutation unless you supply secret values. Push saves configuration only; calibrate new definitions in Grafana. Dry-run checks the manifest locally, so permissions, data source access, and other server checks still apply when you push.

Configure Slack and Alerting notifications

To select a Slack recipient and an Alerting notification policy tree, follow these steps:

Add the following under spec in your complete Watcher manifest. Replace the example IDs and policy tree with your destination values:
YAML
```
notifications:
  slack:
    enabled: true
    targetType: dm
    userId: U_EXAMPLE
    teamId: T_EXAMPLE
    severity: critical
  alerting:
    enabled: true
    severity: critical
    routingTree: example-policy-tree
```
For a Slack channel, use targetType: channel and channelId instead of userId. teamId is optional. Enabled channels require a nonblank channel ID; direct messages require a nonblank user ID even when disabled.
Apply the complete manifest with gcx assistant watchers update WATCHER -f watcher.yaml.
Verify the saved destinations with gcx assistant watchers get WATCHER -o yaml.

Notification severity defaults to warning-and-critical. Set routingTree: "" to use the default notification policy tree. Manage notification policy trees and contact points in Grafana Alerting.

Supply webhook secrets

Use fromEnv or fromFile to supply a secret, for example url: { fromEnv: WATCHER_WEBHOOK_URL }. Exports use { preserve: true } for saved secrets; omitting a secret on update also preserves it. Use { clear: true } to remove one. You can’t use preserve markers when creating a Watcher.

When supplying or clearing a webhook URL, also supply or clear every saved bearer token and signing secret. Keep resolved secrets out of manifests and version control.

Start and pause monitoring

After calibration completes with at least one enabled check, start your Watcher:

gcx assistant watchers start WATCHER

Starting can immediately run a scan; event-only Watchers wait for their trigger. Both start and run can send configured notifications or launch investigations. If start fails, inspect gcx assistant watchers status WATCHER before retrying: monitoring might already be active.

To stop future scans, pause the Watcher:

gcx assistant watchers pause WATCHER

Finish or cancel calibration in Grafana before pausing. An in-flight scan can still finish and send notifications or launch investigations.

Request a run and inspect history

Request a scan without changing the schedule, then inspect recent executions:

gcx assistant watchers run WATCHER -o json
gcx assistant watchers list-runs WATCHER --limit 20 -o json

The run receipt reports acceptance, skipped work, or failure. Acceptance doesn’t mean the scan completed. Its taskId can’t be matched to history IDs; don’t assume the newest execution belongs to your request. Running might require create permission in addition to write permission.

History shows execution status separately from assessment. The default limit is 20, and the maximum is 100. coverage.hasMore indicates additional retained history; results aren’t a snapshot of all past runs. For full run evidence, open the Watcher in Grafana.

Archive and recover a Watcher

Pause before archiving to retain the Watcher’s configuration and history:

gcx assistant watchers pause WATCHER
gcx assistant watchers archive WATCHER

Archive only accepts a Watcher whose status is paused; pause it first. If it is calibrating, finish or cancel calibration in Grafana before pausing. Keep other clients from starting it during this sequence: the pause check and archive aren’t atomic. Archiving can resolve active Grafana Alerting findings.

Recover an archived Watcher without activating monitoring:

gcx assistant watchers unarchive WATCHER

Review its status and automatic-stop deadline before starting again. Recovery retains that deadline. Archive recovery doesn’t restore a previous definition version.

Compare definition versions

To inspect saved definitions before restoring one, follow these steps:

List retained versions:
shell
```
gcx assistant watchers list-versions WATCHER --limit 20 -o json
```
Results contain one page. If coverage.nextBeforeVersion is present, pass it as --before-version to read the next page. Server retention can exclude older versions.

Read the definition you want to inspect:

gcx assistant watchers versions get WATCHER VERSION -o yaml

Compare two definitions:
shell
```
gcx assistant watchers versions diff WATCHER FROM_VERSION TO_VERSION -o json
```
The diff compares definitions only. Add --json path,operation to show changed paths and operations while retaining the compared versions and scope.

Restore a definition version

Restore creates a new current definition version. It requires a non-archived Watcher in draft, paused, or ready state. It retains current notification actions and settings outside the definition; it doesn’t copy historical learned state.

Inspect the Watcher’s lifecycle and current version:
shell
```
gcx assistant watchers status WATCHER -o json
```
For a running Watcher, run gcx assistant watchers pause WATCHER first. Finish or cancel calibration in Grafana before restoring.
Replace CURRENT_VERSION with the version reported by status, then restore the selected VERSION and confirm:
shell
```
gcx assistant watchers versions restore WATCHER VERSION --expected-current-version CURRENT_VERSION -o json
```
A stale guard is rejected. Inspect status and compare again before choosing a new guard.

The restored Watcher stays inactive and can become a draft that needs calibration. You can restore another version from that draft without pausing it. Review it in Grafana before starting monitoring. If the write outcome is unknown, inspect status and list-versions before retrying. A confirmed save followed by failed verification returns exit code 4; inspect the saved version instead of repeating the restore.

Delete a Watcher

To remove a Watcher, run the following command and confirm deletion:

gcx assistant watchers delete WATCHER

Use --force only when you want to skip confirmation.

User guides on Grafana Labs

Dashboards as code

Manage dashboards as code with gcx

Before you begin

Workflow

Explore and modify resources

Explore and modify resources with gcx

Search resources by relevance

Prerequisites

Search for specific resources

Find and delete dashboards using incorrect data sources

Edit remote resources

Manage resources

Manage resources with gcx

Migrate resources between environments

Name pulled files by title

Back up and restore resources

Incident descriptions

Lint resources

Lint resources with gcx

Use the linter

Configure linter rules

Define custom linting rules

Rules for other resources

Diagnose missing telemetry

Diagnose missing telemetry with gcx

Before you begin

Connect without changing your usual context

Log in

Check the overrides

Verify the connection and authentication

Use an agent for diagnostics

Verify a repair and restore the environment

Manage Machine Learning resources

Manage Machine Learning resources

Prerequisites

Inspect resources

Configure the number of records in the output

Apply a resource

Delete a resource

Export and push manifests

Target stack

Push cross-stack

Manage parent alerts

Alert objects

Outlier alerts

Manage OnCall notification rules

Manage OnCall personal notification rules

Authentication and permissions

Manage OnCall notification rules

Rule fields

Routes

Select a step and notification channel

Notification channels

Compare profiles

Compare profiles with gcx

Interpret costs

Share evidence

Manage Assistant Watchers

Manage Assistant Watchers

Before you begin

Create a Watcher and follow calibration

Inspect and update configuration

Manage configuration files

Configure Slack and Alerting notifications

Supply webhook secrets

Start and pause monitoring

Request a run and inspect history

Archive and recover a Watcher

Compare definition versions

Restore a definition version

Delete a Watcher

Manage dashboards as code with `gcx`

Explore and modify resources with `gcx`

Manage resources with `gcx`

Lint resources with `gcx`

Diagnose missing telemetry with `gcx`

Compare profiles with `gcx`