---
title: "Use Assistant Watchers | Grafana Cloud documentation"
description: "Create Assistant Watchers that calibrate checks, scan telemetry, and notify you when services need attention."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Use Assistant Watchers

> Note
> 
> Grafana Assistant Watchers is currently in [public preview](/docs/release-life-cycle/). Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.

Assistant Watchers are always-on agents that check a scoped part of your telemetry on a schedule. You describe what the watcher should monitor, Assistant calibrates concrete checks against your data, and the watcher raises concern when the latest run finds something notable.

Use Watchers for recurring service health checks, noisy-but-important failure modes, and follow-up monitoring after an incident. A watcher can stay quiet when signals look normal, post to Slack when it flags a concern, and launch an Assistant Investigation for critical findings.

## Before you begin

Confirm that your stack and user account can create and run Watchers before you start. Watchers run in the background, so setup affects both the authoring flow and future scheduled runs.

- **Grafana Cloud**: Watchers are a Grafana Cloud-only public preview feature.
- **Grafana Assistant**: Assistant must be enabled for your stack, and any required Assistant terms must be accepted.
- **Data sources**: Select at least one Prometheus or Loki data source for calibration. Assistant can also include Grafana-managed alert checks when they are relevant.
- **Data access**: Watchers run using the creator’s Grafana identity. They can only query data sources and resources that the creator is authorized to access.
- **Permissions**: You need `plugins.app:access` scoped to `plugins:id:grafana-assistant-app` and the relevant `grafana-assistant-app.watcher-agents:*` permissions. Refer to [Manage Assistant access with RBAC](/docs/grafana-cloud/machine-learning/assistant/privacy-and-security/rbac) for details.
- **Slack notifications (optional)**: To post watcher results to Slack, connect Slack for the stack and link your Slack account.
- **Slack thread recalibration (optional)**: To recalibrate Watchers from Slack notification threads, the Grafana app must receive replies in that thread. For channel notifications, add the Grafana app to the channel and mention `@Grafana` in the reply.
- **Investigations (optional)**: To launch investigations from critical watcher findings, you need Assistant investigation access.

## Open Assistant Watchers

Open the Watchers page from the Assistant navigation to view the watcher fleet, create a watcher, and review recent results. The page shows each watcher’s lifecycle status, latest assessment, schedule, and recent activity.

1. Open **Grafana Assistant**.
2. Click **Watchers**.
3. Review the list of existing watchers, or click **New watcher** to create one.

The Watchers list uses these assessment labels:

Expand table

| Label              | Meaning                                                                                                                            |
|--------------------|------------------------------------------------------------------------------------------------------------------------------------|
| **All clear**      | The latest completed run assessed the watched system as OK.                                                                        |
| **Flagged**        | The latest completed run found a warning-level concern.                                                                            |
| **Escalated**      | The latest completed run found a critical concern.                                                                                 |
| **No verdict yet** | The watcher hasn’t completed a run, is still being calibrated, or is in a lifecycle state where the latest verdict isn’t reliable. |

## Create a watcher

Create a watcher by describing the service, system, or failure mode you want Assistant to monitor. The initial form defines the watcher’s intent, data sources, cadence, sensitivity, and optional actions.

1. Open **Grafana Assistant** and go to **Watchers**.
2. Click **New watcher**.
3. Enter a **Name** that makes the watcher easy to recognize later.
4. In **Give the watcher context**, describe what the watcher should monitor. Include important service names, expected rollout behavior, known noise, and acceptable tolerances.
5. Select one or more **Datasources**. Pick the Prometheus and Loki data sources that contain the relevant metrics and logs.
6. Choose how often the watcher **Repeats**. The interval must be between 15 minutes and 1 hour.
7. Choose **Sensitivity** to control how readily the watcher raises concern. Use **Balanced** unless you know the watcher should be more or less sensitive.
8. Configure Slack or investigation actions if you want the watcher to escalate findings automatically.
9. Click **Calibrate**.

Use context that Assistant can’t infer from telemetry alone. For example, tell the watcher whether short error spikes during deployment are expected, which dependencies are noisy, and which symptoms always require attention.

## Calibrate a watcher

Calibration turns your monitoring intent into durable checks that future unattended runs can evaluate. Assistant inspects relevant infrastructure, dashboards, alert rules, and live telemetry, then saves validated queries and a calibration baseline.

During calibration, Assistant asks clarifying questions and shows proposed checks before it finalizes the watcher. It can create PromQL checks, LogQL checks, and alert checks. For Grafana-managed alert checks, Assistant tests Alertmanager label matchers, normalizes the matcher expression, and shows currently firing matches before saving the check. If the alert rule exists only in a datasource-native Alertmanager, use the alert rule’s exact PromQL expression instead. Calibration records normal ranges, warning and critical thresholds, known benign patterns, coverage gaps, and any current conditions that should be treated as already acknowledged.

After calibration finishes, the watcher enters **Ready** state. Review the generated checks and baseline on the watcher detail page before you start scheduled runs.

## Start, pause, and run a watcher

Use lifecycle controls to decide when a watcher actively scans telemetry. A watcher can be ready but inactive, active on a schedule, paused, or run once on demand.

- Click **Start** to turn on scheduled runs.
- Click **Pause** to stop future scheduled runs without deleting the watcher.
- Click **Run now** to trigger one immediate run outside the schedule.
- Click **Delete** to remove the watcher and stop future runs.

Scheduled runs do not overlap for the same watcher. If a run is already in progress, the next run waits or is skipped so the watcher doesn’t scan the same scope concurrently.

## Review watcher results

Review each watcher from its detail page to understand what it checked, what it found, and what state it carried forward. Run history helps you compare assessments over time and inspect the evidence behind a warning or escalation.

The detail page includes:

- The current lifecycle state and latest assessment.
- The calibrated query list, including enabled and disabled checks.
- The calibration baseline that explains normal behavior and thresholds.
- Ongoing issues that the watcher is still tracking.
- Recent runs, including reports, verdict rationale, telemetry evidence, and next-run notes.
- Estimated token consumption based on recent completed runs.
- Creator metadata, including who created the watcher and when it was created.

Open a specific run to review its report and evidence. The run report explains why the watcher chose **All clear**, **Flagged**, or **Escalated**, and the telemetry evidence shows the compact current-versus-baseline data that informed the decision.

## Configure notifications and investigations

Configure actions when you want Watchers to notify teammates or start deeper analysis automatically. Actions are evaluated after a run completes and the watcher decides whether the concern is worth attention.

To post to Slack, enable **Post to Slack** and choose a Slack channel. If Slack isn’t connected, open **Slack settings** from the alert shown in the form, connect the workspace, and link your account.

When a Slack notification makes the wrong call or needs better context, reply in the notification thread to recalibrate the watcher. In a direct message, reply in the thread. In a channel, reply in the thread and mention `@Grafana`; the Grafana app must be a member of that channel.

To launch an investigation, enable **Launch on a critical assessment**. When the watcher escalates a critical finding, Assistant opens a Workspace investigation seeded with the watcher’s findings. Add **Team access** when teammates need to view launched investigations. If you leave team access empty, investigations stay visible only to you and users with sufficient system-created investigation access.

## Understand how Watchers use data

Watchers use the saved calibration output, not the full calibration conversation, when they run later. This keeps scheduled runs bounded and easier to audit.

Each run uses:

- The watcher’s monitoring prompt.
- The calibrated PromQL, LogQL, and alert checks.
- The calibration baseline and thresholds.
- Recent ongoing issues and short-term notes.
- Current telemetry returned by the enabled checks.

Watchers run with the creator’s identity. If the creator loses access to a data source, the watcher may fail or produce incomplete evidence until access is restored or the watcher is recalibrated with available data sources.

Watcher runs consume Assistant tokens, but that usage isn’t billed during public preview. The watcher detail page shows estimated token consumption per run and per hour based on recent completed runs. It also shows the active monthly usage meter for Watchers, including the scope of the limit and the reset date. The meter caps how much Watchers can run rather than charging you for those runs. If the monthly limit is reached, watcher runs are blocked until the reset date. Scheduled Watchers remain active and resume automatically after the limit resets. Tokens used by investigations launched from Watchers are separate from the watcher run estimate.

## Apply best practices

Follow these guidelines to make Watchers useful and reduce noisy notifications.

- Start with a narrow, specific scope, such as one service, namespace, cluster, or failure mode.
- Calibrate against existing alert rules and dashboards where possible, because they often encode trusted thresholds and service ownership.
- Keep checks focused on symptoms that require attention, not every available signal.
- Review generated thresholds and coverage gaps before starting a watcher.
- Use **Run now** after calibration to inspect the first result before enabling scheduled runs.
- Recalibrate after major service changes, dashboard changes, alert threshold changes, or repeated noisy findings.
- Use Slack actions for findings that need human attention, and investigation actions for critical findings that need deeper analysis.

## Troubleshoot common issues

Use these checks when a watcher doesn’t behave as expected. Most issues come from missing permissions, missing data source access, or calibration that no longer matches the monitored system.

### Restore Watchers visibility

Confirm that Watchers is enabled for your Grafana Cloud stack and that you have `grafana-assistant-app.watcher-agents:read` plus `plugins.app:access` scoped to `plugins:id:grafana-assistant-app`. Watchers isn’t available in self-managed Grafana deployments.

### Fix watcher creation

Confirm that you have `grafana-assistant-app.watcher-agents:create`, at least one selected data source, and access to every selected data source. If the create form shows Slack setup warnings, either complete Slack setup or disable Slack posting before saving.

### Finish calibration

Review the calibration conversation for unresolved questions, unsupported data sources, or queries that returned no data. Calibration must save at least one enabled, validated query before a watcher can start.

If an alert matcher check fails, confirm that Grafana-managed Alertmanager is reachable and that the matcher syntax matches the labels on the alert rule. When the rule exists only in a datasource-native Alertmanager, use the rule’s exact PromQL expression instead of Alertmanager label matchers.

### Resume scheduled runs

Confirm the watcher is **Active** and has a valid interval. If the watcher is **Ready**, click **Start**. If it is **Paused**, click **Start** to resume scheduled runs.

### Investigate failed runs

Open the failed run from the watcher detail page and review the error and telemetry evidence. Check whether the creator still has access to the selected data sources, whether a query references labels that no longer exist, and whether the watcher needs recalibration.

If the run failed because the monthly usage limit was reached, wait until the reset date shown on the watcher page. Scheduled Watchers resume automatically after the reset.

### Fix Slack delivery

Confirm Slack is connected for the stack, the creator linked their Slack account, and the selected channel is valid. If channels can’t be loaded, enter the channel ID manually or reopen Slack settings.

## Next steps

- [Manage Assistant access with RBAC](/docs/grafana-cloud/machine-learning/assistant/privacy-and-security/rbac) to grant Watchers permissions.
- [Grafana Assistant Investigations](/docs/grafana-cloud/machine-learning/assistant/platform/investigation) to learn how critical watcher findings can become Workspace investigations.
- [Configure Slack](/docs/grafana-cloud/machine-learning/assistant/configure/slack) if you want Watchers to post notifications to Slack.