Documentation Grafana Cloud Platform Assistant Assistant capabilities Assistant Watchers

Use Assistant Watchers

Assistant Watchers are always-on agents that check a scoped part of your telemetry on a schedule. You describe what the watcher should monitor, Assistant calibrates concrete checks against your data, and the watcher raises concern when the latest run finds something notable.

Use Watchers for recurring service health checks, noisy-but-important failure modes, and follow-up monitoring after an incident. A watcher can stay quiet when signals look normal, notify Slack or a webhook endpoint when it flags a concern, and launch an Assistant Investigation for critical findings.

Before you begin

Confirm that your stack and user account can create and run Watchers before you start. Watchers run in the background, so setup affects both the authoring flow and future scheduled runs.

• Grafana Cloud: Watchers are a Grafana Cloud-only public preview feature.
• Grafana Assistant: Assistant must be enabled for your stack, and any required Assistant terms must be accepted.
• Data sources: Select at least one Prometheus or Loki data source for calibration. Assistant can also include Grafana-managed alert checks when they are relevant.
• Data access: Watchers run using the creator’s Grafana identity. They can only query data sources and resources that the creator is authorized to access.
• Permissions: You need plugins.app:access scoped to plugins:id:grafana-assistant-app and the relevant grafana-assistant-app.watcher-agents:* permissions. Refer to Manage Assistant access with RBAC for details.
• Slack notifications (optional): To post watcher results to Slack, connect Slack for the stack and link your Slack account.
• Webhook notifications (optional): To send watcher notifications to your own systems, you need an HTTPS endpoint that’s reachable from the public internet and accepts POST requests with a JSON body.
• Slack thread recalibration (optional): To recalibrate Watchers from Slack notification threads, the Grafana app must receive replies in that thread. For channel notifications, add the Grafana app to the channel and mention @Grafana in the reply.
• Investigations (optional): To launch investigations from critical watcher findings, you need Assistant investigation access.

Open Assistant Watchers

Open the Watchers page from the Assistant navigation to view the watcher fleet, create a watcher, and review recent results. The page shows each watcher’s lifecycle status, latest assessment, schedule, and recent activity.

1. Open Grafana Assistant.
2. Click Watchers.
3. Review the list of existing watchers, or click New watcher to create one.

The Watchers list uses these assessment labels:

Review your alerting strategy

Use Alerts & Watchers mode in Assistant to review alerting for a service, team, folder, or namespace. Assistant inventories current alert rules and Watchers, checks bounded firing and notification history when it is available, and organizes findings around the failure modes that matter to the scope.

For each failure mode, Assistant recommends an alert, a Watcher, both, dashboard-only coverage, or no notification. The recommendation depends on service maturity, telemetry quality, urgency, operating preferences, and maintenance appetite. For example, a new service can benefit from a flexible, lower-maintenance Watcher while its normal ranges are still changing. A critical service can use a deterministic alert for dependable paging and a Watcher as a second detection and contextual-triage layer.

Assistant explains the evidence and trade-offs, then asks which approach you prefer before changing a Watcher. Alert-rule recommendations are advisory: Assistant can show a concrete expression, duration, labels, and routing diff, but it does not edit the rule. Historical counts are summarized into bounded outcomes rather than loading every Watcher run into the conversation. When history is unavailable, the review identifies the visibility gap instead of inventing noise counts.

Create a watcher

Create a watcher by describing the service, system, or failure mode you want Assistant to monitor. The initial form defines the watcher’s intent, data sources, cadence, sensitivity, and optional actions.

1. Open Grafana Assistant and go to Watchers.
2. Click New watcher.
3. Enter a Name that makes the watcher easy to recognize later.
4. In Give the watcher context, describe what the watcher should monitor. Include important service names, expected rollout behavior, known noise, and acceptable tolerances.
5. Select one or more Datasources. Pick the Prometheus and Loki data sources that contain the relevant metrics and logs.
6. Choose how often the watcher Repeats. The interval must be between 15 minutes and 3 hours.
7. Choose Sensitivity to control how readily the watcher raises concern. Use Balanced unless you know the watcher should be more or less sensitive.
8. Configure Slack, webhook, or investigation actions if you want the watcher to notify teammates or escalate findings automatically.
9. Click Calibrate.

Use context that Assistant can’t infer from telemetry alone. For example, tell the watcher whether short error spikes during deployment are expected, which dependencies are noisy, and which symptoms always require attention.

Calibrate a watcher

Calibration turns your monitoring intent into durable checks that future unattended runs can evaluate. Assistant inspects relevant infrastructure, dashboards, alert rules, and live telemetry, then saves validated queries and a calibration baseline.

During calibration, Assistant asks clarifying questions and shows proposed checks before it finalizes the watcher. It can create PromQL checks, LogQL checks, and alert checks. For Grafana-managed alert checks, Assistant tests Alertmanager label matchers, normalizes the matcher expression, and shows currently firing matches before saving the check. If the alert rule exists only in a datasource-native Alertmanager, use the alert rule’s exact PromQL expression instead. Calibration records normal ranges, warning and critical thresholds, known benign patterns, coverage gaps, and any current conditions that should be treated as already acknowledged.

After calibration finishes, the watcher enters Ready state. Review the generated checks and baseline on the watcher detail page before you start scheduled runs.

Start, pause, and run a watcher

Use lifecycle controls to decide when a watcher actively scans telemetry. A watcher can be ready but inactive, active on a schedule, paused, or run once on demand.

• Click Start to turn on scheduled runs.
• Click Pause to stop future scheduled runs without deleting the watcher.
• Click Run now to trigger one immediate run outside the schedule.
• Click Delete to remove the watcher and stop future runs.

Scheduled runs do not overlap for the same watcher. If a run is already in progress, the next run waits or is skipped so the watcher doesn’t scan the same scope concurrently.

Review watcher results

Review each watcher from its detail page to understand what it checked, what it found, and what state it carried forward. Run history helps you compare assessments over time and inspect the evidence behind a warning or escalation.

The detail page includes:

• The current lifecycle state and latest assessment.
• The calibrated query list, including enabled and disabled checks.
• The calibration baseline that explains normal behavior and thresholds.
• Ongoing issues that the watcher is still tracking.
• Recent runs, including reports, verdict rationale, telemetry evidence, and next-run notes.
• Estimated token consumption based on recent completed runs.
• Creator metadata, including who created the watcher and when it was created.

Open a specific run to review its report and evidence. The run report explains why the watcher chose All clear, Flagged, or Escalated, and the telemetry evidence shows the compact current-versus-baseline data that informed the decision.

Configure notifications and investigations

Configure actions when you want Watchers to notify teammates or start deeper analysis automatically. Actions are evaluated after a run completes and the watcher decides whether the concern is worth attention.

Each run makes one notification decision. Slack and webhook destinations can be enabled together, individually, or not at all, and each destination applies its own Notify for severity filter to that shared decision. Configure notifications when you create the watcher or later from the watcher detail page.

For each destination, choose Warnings and critical issues (the default) or Critical issues. Critical issue recovery updates bypass this filter, even for a destination set to Critical issues.

Understand when Watchers notify

Watchers suppress repeat notifications so a persistent issue doesn’t page you on every run. A run sends a notification only when the watcher judges the situation worth reporting and at least one of these changes occurred:

• A new issue appears.
• An issue last notified as a warning escalates to critical.
• A previously notified critical issue is still open and enough time has passed since the last notification, about four hours by default.
• A previously notified critical issue genuinely resolves.

Warning reminders and warning resolutions are silent by default. A run with no eligible change stays quiet, even when previously reported issues remain open. Enabling a new destination doesn’t replay earlier notifications; the destination receives notifications starting from the next eligible run.

Post to Slack

To post to Slack, enable Post to Slack and choose a Slack channel. If Slack isn’t connected, open Slack settings from the alert shown in the form, connect the workspace, and link your account.

When a Slack notification makes the wrong call or needs better context, reply in the notification thread to recalibrate the watcher. In a direct message, reply in the thread. In a channel, reply in the thread and mention @Grafana; the Grafana app must be a member of that channel.

Send notifications to a webhook

To send watcher notifications to your own tooling, for example an incident management system, an event bus, or an internal service, enable Send to a webhook and enter the endpoint URL. Optionally, click Add authentication to configure:

• Bearer token: sent with every delivery in the Authorization header as Bearer <TOKEN>.
• HMAC signing secret: signs every delivery so your endpoint can verify that the request came from Grafana Assistant. Refer to Verify webhook signatures.

The endpoint URL must use HTTPS, resolve to a publicly reachable host, and must not include a username or password. Only ports 443 and 80 are accepted, and IPv6 address literals aren’t supported. Assistant treats the URL, bearer token, and signing secret as secrets: they’re stored encrypted, they’re never returned by the API or the UI after saving, and only a sanitized host preview is shown. To change a saved secret, enter a new value or reset it; you can’t read the existing one back.

Changing Notify for applies to future notification decisions. Unlike an endpoint or credential edit, it doesn’t cancel deliveries that are already queued.

Receive webhook deliveries

Assistant delivers each notification as an HTTPS POST request with a JSON body. Respond with any 2xx status code to acknowledge the delivery. Each request includes these headers:

The body is versioned and contains an operational summary. It has no dedicated fields for query expressions, log lines, or full telemetry digests, though notification.message can include summarized telemetry values. The following example shows a critical notification from a watcher that also launched an investigation:

JSON Copy

{
  "version": "1",
  "type": "watcher.notification",
  "notificationId": "0b9e4a83-6f2d-5c3e-9d41-a1b2c3d4e5f6",
  "timestamp": "2026-08-04T14:30:00Z",
  "watcher": {
    "id": "4f8c1a2b-3d4e-5f60-7a8b-9c0d1e2f3a4b",
    "name": "Checkout health",
    "url": "https://example.grafana.net/a/grafana-assistant-app/watchers/4f8c1a2b-3d4e-5f60-7a8b-9c0d1e2f3a4b"
  },
  "run": {
    "id": "7d6e5f4a-3b2c-1d0e-9f8a-7b6c5d4e3f2a",
    "assessment": "critical"
  },
  "notification": {
    "assessment": "critical",
    "message": "Checkout errors are elevated above the calibrated baseline.",
    "ongoingCritical": 1,
    "ongoingWarning": 0
  },
  "investigation": {
    "id": "2a1b0c9d-8e7f-6a5b-4c3d-2e1f0a9b8c7d",
    "url": "https://example.grafana.net/a/grafana-assistant-app/investigations/2a1b0c9d-8e7f-6a5b-4c3d-2e1f0a9b8c7d"
  }
}

The notification.assessment field reflects the severity of what changed in this run: critical for critical findings, warning for warning-level findings, and ok when the notification announces a resolved critical issue. The investigation object is present only when the run launched an investigation. Payload version 1 evolves by adding fields; treat unknown fields as forward-compatible additions.

Expect these delivery characteristics:

• A notification can be delivered more than once, or not at all. A retry can arrive after your endpoint already processed the request, so deduplicate by notificationId; a notification that fails all delivery attempts is missed rather than queued indefinitely. Redirects aren’t followed.
• The body and idempotency key are identical across retries. The signature timestamp can change between attempts.
• Notifications aren’t guaranteed to arrive in the order their runs completed.
• Changing the webhook configuration cancels deliveries that are still pending, so a notification accepted for one endpoint or credential is never sent to a newly configured one.

Verify webhook signatures

When you configure an HMAC signing secret, each webhook request includes the Unix timestamp in X-Assistant-Signature-Timestamp and a lowercase hexadecimal HMAC-SHA256 digest in X-Assistant-Signature. Grafana Assistant calculates the digest as:

text Copy

hex(HMAC-SHA256(<SECRET>, <TIMESTAMP> + ":" + <RAW_REQUEST_BODY>))

Use the request body bytes exactly as received, compare the digest in constant time, and reject timestamps outside an acceptable window to limit replay attacks.

Launch investigations

To launch an investigation, enable Launch on a critical assessment. When the watcher escalates a critical finding, Assistant opens a Workspace investigation seeded with the watcher’s findings. Add Team access when teammates need to view launched investigations. If you leave team access empty, investigations stay visible only to you and users with sufficient system-created investigation access.

Understand how Watchers use data

Watchers use the saved calibration output, not the full calibration conversation, when they run later. This keeps scheduled runs bounded and easier to audit.

Each run uses:

• The watcher’s monitoring prompt.
• The calibrated PromQL, LogQL, and alert checks.
• The calibration baseline and thresholds.
• Recent ongoing issues and short-term notes.
• Current telemetry returned by the enabled checks.

Watchers run with the creator’s identity. If the creator loses access to a data source, the watcher may fail or produce incomplete evidence until access is restored or the watcher is recalibrated with available data sources.

Watcher runs consume Assistant tokens, but that usage isn’t billed during public preview. The watcher detail page shows estimated token consumption per run and per hour based on recent completed runs. It also shows the active monthly usage meter for Watchers, including the scope of the limit and the reset date. The meter caps how much Watchers can run rather than charging you for those runs. If the monthly limit is reached, watcher runs are blocked until the reset date. Scheduled Watchers remain active and resume automatically after the limit resets. Tokens used by investigations launched from Watchers are separate from the watcher run estimate.

Apply best practices

Follow these guidelines to make Watchers useful and reduce noisy notifications.

• Start with a narrow, specific scope, such as one service, namespace, cluster, or failure mode.
• Calibrate against existing alert rules and dashboards where possible, because they often encode trusted thresholds and service ownership.
• Keep checks focused on symptoms that require attention, not every available signal.
• Review generated thresholds and coverage gaps before starting a watcher.
• Use Run now after calibration to inspect the first result before enabling scheduled runs.
• Recalibrate after major service changes, dashboard changes, alert threshold changes, or repeated noisy findings.
• Use Slack actions for findings that need human attention, webhook actions to feed notifications into your own tooling, and investigation actions for critical findings that need deeper analysis.

Troubleshoot common issues

Use these checks when a watcher doesn’t behave as expected. Most issues come from missing permissions, missing data source access, or calibration that no longer matches the monitored system.

Restore Watchers visibility

Confirm that Watchers is enabled for your Grafana Cloud stack and that you have grafana-assistant-app.watcher-agents:read plus plugins.app:access scoped to plugins:id:grafana-assistant-app. Watchers isn’t available in self-managed Grafana deployments.

Fix watcher creation

Confirm that you have grafana-assistant-app.watcher-agents:create, at least one selected data source, and access to every selected data source. If the create form shows Slack setup warnings, either complete Slack setup or disable Slack posting before saving.

Finish calibration

Review the calibration conversation for unresolved questions, unsupported data sources, or queries that returned no data. Calibration must save at least one enabled, validated query before a watcher can start.

If an alert matcher check fails, confirm that Grafana-managed Alertmanager is reachable and that the matcher syntax matches the labels on the alert rule. When the rule exists only in a datasource-native Alertmanager, use the rule’s exact PromQL expression instead of Alertmanager label matchers.

Resume scheduled runs

Confirm the watcher is Active and has a valid interval. If the watcher is Ready, click Start. If it is Paused, click Start to resume scheduled runs.

Investigate failed runs

Open the failed run from the watcher detail page and review the error and telemetry evidence. Check whether the creator still has access to the selected data sources, whether a query references labels that no longer exist, and whether the watcher needs recalibration.

If the run failed because the monthly usage limit was reached, wait until the reset date shown on the watcher page. Scheduled Watchers resume automatically after the reset.

Fix Slack delivery

Confirm Slack is connected for the stack, the creator linked their Slack account, and the selected channel is valid. If channels can’t be loaded, enter the channel ID manually or reopen Slack settings.

Fix webhook delivery

Confirm the webhook is enabled and the endpoint responds with a 2xx status within 10 seconds, directly at the configured URL, over HTTPS with a publicly trusted TLS certificate. Assistant retries timeouts, connection failures, and 408 or 5xx responses for up to three total attempts; redirects, TLS validation failures, and 4xx responses other than 408 fail immediately.

Configuration edits cancel deliveries that were pending during the edit; new notifications use the updated configuration. If your endpoint rejects deliveries during signature verification, confirm it computes the digest over the raw request body bytes and the timestamp from X-Assistant-Signature-Timestamp, and update it after you reset a secret in Grafana. If a notification seems missing, check whether the run made an eligible change; refer to Understand when Watchers notify.

Next steps

• Manage Assistant access with RBAC to grant Watchers permissions.
• Grafana Assistant Investigations to learn how critical watcher findings can become Workspace investigations.
• Configure Slack if you want Watchers to post notifications to Slack.