Documentation Grafana Cloud AI and machine learning Assistant Assistant capabilities Assistant Watchers

Use Assistant Watchers

Assistant Watchers are always-on agents that check a scoped part of your telemetry on a schedule. You describe what the watcher should monitor, Assistant calibrates concrete checks against your data, and the watcher raises concern when the latest run finds something notable.

Use Watchers for recurring service health checks, noisy-but-important failure modes, and follow-up monitoring after an incident. A watcher can stay quiet when signals look normal, post to Slack when it flags a concern, and launch an Assistant Investigation for critical findings.

Before you begin

Confirm that your stack and user account can create and run Watchers before you start. Watchers run in the background, so setup affects both the authoring flow and future scheduled runs.

• Grafana Cloud: Watchers are a Grafana Cloud-only public preview feature.
• Grafana Assistant: Assistant must be enabled for your stack, and any required Assistant terms must be accepted.
• Data sources: Select at least one Prometheus or Loki data source for calibration. Assistant can also include Grafana-managed alert checks when they are relevant.
• Data access: Watchers run using the creator’s Grafana identity. They can only query data sources and resources that the creator is authorized to access.
• Permissions: You need plugins.app:access scoped to plugins:id:grafana-assistant-app and the relevant grafana-assistant-app.watcher-agents:* permissions. Refer to Manage Assistant access with RBAC for details.
• Slack notifications (optional): To post watcher results to Slack, connect Slack for the stack and link your Slack account.
• Slack thread recalibration (optional): To recalibrate Watchers from Slack notification threads, the Grafana app must receive replies in that thread. For channel notifications, add the Grafana app to the channel and mention @Grafana in the reply.
• Investigations (optional): To launch investigations from critical watcher findings, you need Assistant investigation access.

Open Assistant Watchers

Open the Watchers page from the Assistant navigation to view the watcher fleet, create a watcher, and review recent results. The page shows each watcher’s lifecycle status, latest assessment, schedule, and recent activity.

1. Open Grafana Assistant.
2. Click Watchers.
3. Review the list of existing watchers, or click New watcher to create one.

The Watchers list uses these assessment labels:

Create a watcher

Create a watcher by describing the service, system, or failure mode you want Assistant to monitor. The initial form defines the watcher’s intent, data sources, cadence, sensitivity, and optional actions.

1. Open Grafana Assistant and go to Watchers.
2. Click New watcher.
3. Enter a Name that makes the watcher easy to recognize later.
4. In Give the watcher context, describe what the watcher should monitor. Include important service names, expected rollout behavior, known noise, and acceptable tolerances.
5. Select one or more Datasources. Pick the Prometheus and Loki data sources that contain the relevant metrics and logs.
6. Choose how often the watcher Repeats. The interval must be between 15 minutes and 1 hour.
7. Choose Sensitivity to control how readily the watcher raises concern. Use Balanced unless you know the watcher should be more or less sensitive.
8. Configure Slack or investigation actions if you want the watcher to escalate findings automatically.
9. Click Calibrate.

Use context that Assistant can’t infer from telemetry alone. For example, tell the watcher whether short error spikes during deployment are expected, which dependencies are noisy, and which symptoms always require attention.

Calibrate a watcher

Calibration turns your monitoring intent into durable checks that future unattended runs can evaluate. Assistant inspects relevant infrastructure, dashboards, alert rules, and live telemetry, then saves validated queries and a calibration baseline.

During calibration, Assistant asks clarifying questions and shows proposed checks before it finalizes the watcher. It can create PromQL checks, LogQL checks, and alert checks. For Grafana-managed alert checks, Assistant tests Alertmanager label matchers, normalizes the matcher expression, and shows currently firing matches before saving the check. If the alert rule exists only in a datasource-native Alertmanager, use the alert rule’s exact PromQL expression instead. Calibration records normal ranges, warning and critical thresholds, known benign patterns, coverage gaps, and any current conditions that should be treated as already acknowledged.

After calibration finishes, the watcher enters Ready state. Review the generated checks and baseline on the watcher detail page before you start scheduled runs.

Start, pause, and run a watcher

Use lifecycle controls to decide when a watcher actively scans telemetry. A watcher can be ready but inactive, active on a schedule, paused, or run once on demand.

• Click Start to turn on scheduled runs.
• Click Pause to stop future scheduled runs without deleting the watcher.
• Click Run now to trigger one immediate run outside the schedule.
• Click Delete to remove the watcher and stop future runs.

Scheduled runs do not overlap for the same watcher. If a run is already in progress, the next run waits or is skipped so the watcher doesn’t scan the same scope concurrently.

Review watcher results

Review each watcher from its detail page to understand what it checked, what it found, and what state it carried forward. Run history helps you compare assessments over time and inspect the evidence behind a warning or escalation.

The detail page includes:

• The current lifecycle state and latest assessment.
• The calibrated query list, including enabled and disabled checks.
• The calibration baseline that explains normal behavior and thresholds.
• Ongoing issues that the watcher is still tracking.
• Recent runs, including reports, verdict rationale, telemetry evidence, and next-run notes.
• Estimated token consumption based on recent completed runs.
• Creator metadata, including who created the watcher and when it was created.

Open a specific run to review its report and evidence. The run report explains why the watcher chose All clear, Flagged, or Escalated, and the telemetry evidence shows the compact current-versus-baseline data that informed the decision.

Configure notifications and investigations

Configure actions when you want Watchers to notify teammates or start deeper analysis automatically. Actions are evaluated after a run completes and the watcher decides whether the concern is worth attention.

To post to Slack, enable Post to Slack and choose a Slack channel. If Slack isn’t connected, open Slack settings from the alert shown in the form, connect the workspace, and link your account.

When a Slack notification makes the wrong call or needs better context, reply in the notification thread to recalibrate the watcher. In a direct message, reply in the thread. In a channel, reply in the thread and mention @Grafana; the Grafana app must be a member of that channel.

To launch an investigation, enable Launch on a critical assessment. When the watcher escalates a critical finding, Assistant opens a Workspace investigation seeded with the watcher’s findings. Add Team access when teammates need to view launched investigations. If you leave team access empty, investigations stay visible only to you and users with sufficient system-created investigation access.

Understand how Watchers use data

Watchers use the saved calibration output, not the full calibration conversation, when they run later. This keeps scheduled runs bounded and easier to audit.

Each run uses:

• The watcher’s monitoring prompt.
• The calibrated PromQL, LogQL, and alert checks.
• The calibration baseline and thresholds.
• Recent ongoing issues and short-term notes.
• Current telemetry returned by the enabled checks.

Watchers run with the creator’s identity. If the creator loses access to a data source, the watcher may fail or produce incomplete evidence until access is restored or the watcher is recalibrated with available data sources.

Watcher runs consume Assistant tokens, but that usage isn’t billed during public preview. The watcher detail page shows estimated token consumption per run and per hour based on recent completed runs. It also shows the active monthly usage meter for Watchers, including the scope of the limit and the reset date. The meter caps how much Watchers can run rather than charging you for those runs. If the monthly limit is reached, watcher runs are blocked until the reset date. Scheduled Watchers remain active and resume automatically after the limit resets. Tokens used by investigations launched from Watchers are separate from the watcher run estimate.

Apply best practices

Follow these guidelines to make Watchers useful and reduce noisy notifications.

• Start with a narrow, specific scope, such as one service, namespace, cluster, or failure mode.
• Calibrate against existing alert rules and dashboards where possible, because they often encode trusted thresholds and service ownership.
• Keep checks focused on symptoms that require attention, not every available signal.
• Review generated thresholds and coverage gaps before starting a watcher.
• Use Run now after calibration to inspect the first result before enabling scheduled runs.
• Recalibrate after major service changes, dashboard changes, alert threshold changes, or repeated noisy findings.
• Use Slack actions for findings that need human attention, and investigation actions for critical findings that need deeper analysis.

Troubleshoot common issues

Use these checks when a watcher doesn’t behave as expected. Most issues come from missing permissions, missing data source access, or calibration that no longer matches the monitored system.

Restore Watchers visibility

Confirm that Watchers is enabled for your Grafana Cloud stack and that you have grafana-assistant-app.watcher-agents:read plus plugins.app:access scoped to plugins:id:grafana-assistant-app. Watchers isn’t available in self-managed Grafana deployments.

Fix watcher creation

Confirm that you have grafana-assistant-app.watcher-agents:create, at least one selected data source, and access to every selected data source. If the create form shows Slack setup warnings, either complete Slack setup or disable Slack posting before saving.

Finish calibration

Review the calibration conversation for unresolved questions, unsupported data sources, or queries that returned no data. Calibration must save at least one enabled, validated query before a watcher can start.

If an alert matcher check fails, confirm that Grafana-managed Alertmanager is reachable and that the matcher syntax matches the labels on the alert rule. When the rule exists only in a datasource-native Alertmanager, use the rule’s exact PromQL expression instead of Alertmanager label matchers.

Resume scheduled runs

Confirm the watcher is Active and has a valid interval. If the watcher is Ready, click Start. If it is Paused, click Start to resume scheduled runs.

Investigate failed runs

Open the failed run from the watcher detail page and review the error and telemetry evidence. Check whether the creator still has access to the selected data sources, whether a query references labels that no longer exist, and whether the watcher needs recalibration.

If the run failed because the monthly usage limit was reached, wait until the reset date shown on the watcher page. Scheduled Watchers resume automatically after the reset.

Fix Slack delivery

Confirm Slack is connected for the stack, the creator linked their Slack account, and the selected channel is valid. If channels can’t be loaded, enter the channel ID manually or reopen Slack settings.

Next steps

• Manage Assistant access with RBAC to grant Watchers permissions.
• Grafana Assistant Investigations to learn how critical watcher findings can become Workspace investigations.
• Configure Slack if you want Watchers to post notifications to Slack.