User guides on Grafana Labs

Dashboards as code

Mon, 12 Oct 2026 15:20:54 +0000

Manage dashboards as code with `gcx`

With this workflow, you can define and manage dashboards as code, saving them to a version control system like Git. This is useful for teams that want to maintain a history of changes, collaborate on dashboard design, and ensure consistency across environments.

Before you begin

In order to use the gcx dev serve functionality, you need to enable a feature toggle in your configuration file:

[feature_toggles]
kubernetesDashboards = true

Refer to Configure Grafana to learn more about enabling feature toggles.

Although the local preview proxy supports service account token, Basic, and mTLS contexts, use a service account token instead of OAuth for gcx dev serve. OAuth is rejected before startup until refresh-token rotation can be persisted safely by the long-running server.

Workflow

To manage dashboards as code with gcx follow these steps:

Use a dashboard generation script (for example, with the Foundation SDK). You can find an example implementation in the Grafana as code hands-on lab repository.

Serve and preview the output of the dashboard generator locally:

gcx config use-context YOUR_CONTEXT  # for example "dev"
gcx dev serve --script 'go run scripts/generate-dashboard.go' --watch './scripts'

When the output looks correct, generate dashboard manifest files:

go run scripts/generate-dashboard.go --generate-resource-manifests --output './resources'

Push the generated resources to your Grafana instance:

gcx config use-context YOUR_CONTEXT  # for example "dev"
gcx resources push -p ./resources/

Explore and modify resources

Mon, 12 Oct 2026 15:20:54 +0000

Explore and modify resources with `gcx`

The Grafana CLI allows you to interact with Grafana resources directly from your terminal. You can browse, inspect, update, and delete resources without using the Grafana UI.

Use this approach if you’re an advanced user and want to manage resources more efficiently, or to integrate Grafana operations into automated workflows.

Find and delete dashboards using incorrect data sources

To identify and remove production dashboards relying on non-production data sources, use the command below to list dashboard UIDs along with the data source UIDs used in their panels:

gcx resources get dashboards --context prod | jq '.items | map({ uid: .metadata.name, datasources: .spec.panels | map(.datasource.uid)  })'
[
   {
      "uid": "important-production-dashboard",
      "datasources": [
         "mimir-prod"
      ]
   },
   {
      "uid": "test-dashboard-from-dev",
      "datasources": [
         "mimir-dev"
      ]
   },
   {
      "uid": "test-dashboard-from-stg",
      "datasources": [
         "mimir-stg"
      ]
   }
]

Next, identify the dashboards that are using unexpected data sources, and delete them:

gcx resources delete dashboards/test-dashboard-from-stg,test-dashboard-from-dev
✔ 2 resources deleted, 0 errors

Edit remote resources

You can edit resources directly from the default editor, without having to pull them first:

gcx resources edit dashboard/edit-me-please

This command opens the default editor as configured by the EDITOR environment variable, or fall back to ‘vi’ for Linux or ’notepad’ for Windows.

After the editor process terminates, the resource is updated in the Grafana instance targeted by the current context. Note that edition will be cancelled if no changes are written to the file or if the file after edition is empty.

Manage resources

Mon, 12 Oct 2026 15:20:54 +0000

Manage resources with `gcx`

Migrate resources between environments

Grafana resources can be migrated from one environment to another, for example from a development to production environment.

You need to configure one context per environment. In this example scenario, use dev for the development environment and prod for production.

Make changes to dashboards and other resources using the Grafana UI in your development instance.
Pull those resources from the development environment to your local machine:
shell
```
gcx resources pull --context dev # Add `-o yaml` export resources as YAML 
```
Push the resources to production:
shell
```
gcx resources push --context prod
```

Note
Resources are pulled and pushed from the ./resources directory by default. Configure this path with the --path/-p flags.

Resources are pulled and pushed from the ./resources directory by default. Configure this path with the --path/-p flags.

Back up and restore resources

Follow this workflow to back up all Grafana resources from one instance and later restore them. This can be useful to replicate a configuration or perform disaster recovery.

Ensure the current context points to the Grafana instance to backup/restore:
shell
```
gcx config use-context YOUR_CONTEXT  # for example "prod"
```

Pull all resources from your target environment:

gcx resources pull --path ./backup-prod # Add `-o yaml` to export resources as YAML

Save the exported resources to version control or cloud storage.
Push the resources to restore them:
shell
```
gcx resources push --path ./backup-prod
```

Lint resources

Mon, 12 Oct 2026 15:20:54 +0000

Lint resources with `gcx`

Use the Grafana CLI linter to verify that resources, such as dashboards and alert rules, comply with good practices and environment-specific policies.

Use the linter

To lint resources run:

gcx dev lint run ./resources

The following applies:

Directories are recursively explored.
All built-in rules are enabled by default.

Configure linter rules

For a finer control, you can configure the linter rules:

# Disable all rules for a resource type:
gcx dev lint run --disable-resource dashboard ./resources

# Disable all rules in a category:
gcx dev lint run --disable-category idiomatic ./resources

# Disable specific rules:
gcx dev lint run --disable uneditable-dashboard --disable panel-title-description ./resources

# Enable rules for specific resource types:
gcx dev lint run --disable-all --enable-resource dashboard ./resources

# Enable only some categories:
gcx dev lint run --disable-all --enable-category idiomatic ./resources

# Enable only specific rules:
gcx dev lint run --disable-all --enable uneditable-dashboard ./resources

You can modify the severity level of a rule by updating the custom.severity annotation. Valid values are warning and error.

Define custom linting rules

Custom and built-in rules are defined in Rego, the policy language used by Open Policy Agent (OPA). This ensures that resources comply with policies specific to your environment.

You can scaffold new custom rules with gcx:

# Creates a new "dashboard" linter rule in the current directory:
gcx dev lint new dashboard custom-rule

This generates a file with the bootstrapped rule:

# METADATA
# description: Briefly describe the rule here.
# custom:
#  severity: warning
package custom.gcx.rules.dashboard.idiomatic["custom-rule"]

import data.gcx.result
import data.gcx.utils

# Dashboard v1
report contains violation if {
	utils.resource_is_dashboard_v1(input)

	input.spec.timezone != "utc"

	violation := result.fail(rego.metadata.chain(), sprintf("timezone is '%s', expected 'utc'", input.spec.timezone))
}

# Dashboard v2
report contains violation if {
	utils.resource_is_dashboard_v2(input)

	input.spec.timeSettings.timezone != "utc"

	violation := result.fail(rego.metadata.chain(), sprintf("timezone is '%s', expected 'utc'", input.spec.timeSettings.timezone))
}

Refer to the existing Built-in rules for inspiration to write custom ones.

Rules for other resources

You can create rules for other resource types than dashboards, or in categories other than idiomatic:

# Creates a new "alertrule" linter rule, categorized under "bug":
gcx dev lint new alertrule custom-rule -c bug

Diagnose missing telemetry

Mon, 12 Oct 2026 15:20:54 +0000

Diagnose missing telemetry with `gcx`

If you’re investigating an empty dashboard or missing application telemetry, use this guide to learn how to use gcx to gather evidence before proposing a repair. You don’t need a new diagnostic skill or Grafana Cloud.

You can investigate directly or use an agent with gcx existing skills. Start with evidence, ask before repairs, and verify the original symptom afterward. If you use an agent, its harness and model access are separate from gcx.

Before you begin

Before you begin, make sure to:

Follow Install gcx if gcx version is unavailable.
Follow Install Agent Skills for your harness. Start a session that can discover the installed skills. The existing debug-with-grafana skill handles investigation; setup-gcx helps with setup.
Use a reachable Grafana instance supported by your gcx version, with access to the relevant datasources. A static instrumentation check alone does not provide a telemetry destination.
If you have no backend, start docker-otel-lgtm for a local development sink. Setting up a sink does not authorize changing an existing application’s exporter. A local proof does not establish that your production destination works.
Know which application or fixture you may inspect. Do not enumerate unrelated containers, environments, or services to compensate for missing context.

The command examples use the current gcx command surface. Check gcx version and the relevant command’s --help when following them with another release.

Connect without changing your usual context

If you have already configured the destination, use explicit --config and --context arguments. Don’t switch the current context just for this investigation. Otherwise, create a separate private configuration outside your checkout.

The examples use GCX_CONFIG, the actual gcx environment override. If it’s pointing at another configuration, unset it first:

GCX_DIAGNOSTICS_DIR=$(mktemp -d)
export GCX_CONFIG="$GCX_DIAGNOSTICS_DIR/config.yaml"
(umask 077; printf '{}\n' > "$GCX_CONFIG")

Keep the path until cleanup. In another terminal, set the variable to the same path; don’t create a second empty configuration by repeating the block.

Log in

For an existing Grafana destination, use interactive login with a new context in that file, replacing the example URL:

gcx login diagnostics --config "$GCX_CONFIG" \
  --server https://your-grafana.example

For local LGTM, use that repository’s local connection instructions instead of this login step. Refer to Configure gcx for supported authentication methods. Supply credentials locally through the supported authentication flow, not in chat, copied transcripts, or committed files. Prefer read-only access when available as approval instructions don’t restrict API permissions.

Check the overrides

Environment overrides can still affect the selected configuration. Review which overrides you intentionally set before running queries and don’t dump your whole environment or configuration into a transcript. Don’t disable TLS verification to make a connection error disappear.

Use your selected context name below, where diagnostics is the login example:

gcx config check --config "$GCX_CONFIG" --context diagnostics
gcx datasources list --config "$GCX_CONFIG" --context diagnostics

Verify the connection and authentication

Stop and resolve any connection or authentication errors before interpreting query results.

For example, against a Prometheus datasource:

gcx metrics query 'vector(1)' --datasource 'PROMETHEUS_DATASOURCE_UID' \
  --error-on-empty \
  --config "$GCX_CONFIG" --context diagnostics

Keep in mind the following:

Choose the datasource UID returned by discovery, rather than assuming a default.
A successful configuration check doesn’t prove application ingestion.
A returned value proves that query path works, not that the application emitted metrics.

With --error-on-empty, an empty response exits unsuccessfully; without it, a successful empty query, an invalid query, and a failed connection are different observations. The flag is supported on the Prometheus query, Loki logs/metrics, Tempo search/metrics, and Pyroscope query commands, but it’s not a generic flag for unrelated datasource query commands.

Use an agent for diagnostics

Start the agent in the relevant application or test checkout, and provide it with the symptoms and boundaries for diagnostics:

The original dashboard URL, failed test assertion, or checker finding.
The explicit gcx config path, context, and intended destination (no secrets).
The application/service identity and one representative operation, if known.
The UTC time window, observed query, and request/trace identifiers, if available.
The relevant Compose project, deployment, or Collector configuration you own.
Which resources may be inspected and which actions require approval.

Make sure the agent doesn’t invent answers and to identify what information or access is missing. For example, use this prompt, filling in what you know:

Use the existing gcx skills to investigate `<symptom>`. Query only `<config path and context>` and inspect only `<application/fixture scope>`. The expected operation is `<operation>` in `<time window>`. Diagnose first and distinguish observed evidence from hypotheses. State what you could not observe; do not infer data loss from missing access. Ask before changing application, Collector, dashboard, or test configuration, restarting resources, or enabling payload logging. Do not weaken an assertion to make a test pass. Propose the smallest justified repair, and stop for approval.

An empty dashboard can result from no data being created, export failure, filtering, an incorrect query, or a time/identity mismatch. Static configuration and absence of results are leads, not proof of a particular failure boundary.

Always ask for the evidence supporting each conclusion and the next discriminating check.

Verify a repair and restore the environment

After approving a specific change:

Exercise a fresh representative operation and record its time and available identifiers. Old telemetry must not satisfy the recovery check.
Query the relevant datasource, then recheck the original dashboard or rerun the original test uncached. A successful backend query alone does not prove the panel or assertion is correct.
Review the actual diff and runtime changes. Functional recovery doesn’t prove that unrelated dashboard or configuration fields were preserved.
Restore temporary diagnostic configuration and logging. Remove temporary backups and stop only resources created for this investigation, after confirming they are no longer needed. Never clean up pre-existing resources by assumption.
Record the confirmed cause, remaining unknowns, approved changes, and result.

Debug-exporter payloads and logs may contain sensitive data. Enable them only with permission, for a bounded investigation, and never publish raw output. Avoid full environment dumps or HTTP payload logging that can expose credentials.

If you created the private directory above, inspect it locally and remove only that directory when finished. Remove any diagnostic credentials stored by login using your normal credential-management process; deleting a config file doesn’t revoke a server-side token. Explicit configuration or context arguments leave your usual current context unchanged. Don’t delete an existing configuration used for the investigation, or revoke a credential shared with another workflow.

User guides on Grafana Labs

Dashboards as code

Manage dashboards as code with gcx

Before you begin

Workflow

Explore and modify resources

Explore and modify resources with gcx

Find and delete dashboards using incorrect data sources

Edit remote resources

Manage resources

Manage resources with gcx

Migrate resources between environments

Back up and restore resources

Lint resources

Lint resources with gcx

Use the linter

Configure linter rules

Define custom linting rules

Rules for other resources

Diagnose missing telemetry

Diagnose missing telemetry with gcx

Before you begin

Connect without changing your usual context

Log in

Check the overrides

Verify the connection and authentication

Use an agent for diagnostics

Verify a repair and restore the environment

Manage dashboards as code with `gcx`

Explore and modify resources with `gcx`

Manage resources with `gcx`

Lint resources with `gcx`

Diagnose missing telemetry with `gcx`