Documentation Grafana Cloud Platform Assistant Assistant capabilities Assistant Watchers

Use Assistant Watchers

Grafana Assistant Watchers check your telemetry on a schedule and report notable changes. Describe what to monitor, let Assistant calibrate checks against your data, then review findings and adjust the watcher as your service changes. Turn on automatic recalibration to let Assistant review and update its checks on a schedule.

Use Watchers for service health checks and follow-up monitoring after an incident. Choose Slack, Microsoft Teams, webhooks, or Grafana Alerting for notifications, and optionally launch an Assistant Investigation for critical findings.

Before you begin

To create and run a watcher, you need:

• A Grafana Cloud stack with Grafana Assistant enabled and any required Assistant terms accepted.
• Permission to open Watchers, create a watcher, and calibrate it. Creating a watcher from the form requires both grafana-assistant-app.watcher-agents:create and grafana-assistant-app.watcher-agents:write. Refer to Manage Assistant access with RBAC for the required permissions.
• At least one supported data source that you can read and query: Prometheus, Amazon Managed Service for Prometheus, Loki, BigQuery, Infinity, CloudWatch, Synthetic Monitoring, Tempo, or Pyroscope. Data sources with team label-based access control (LBAC) rules aren’t supported.

Open Assistant Watchers

Open the Watchers page to review existing watchers or create one.

1. In the main menu, expand AI.
2. Click Watchers.

Create a watcher

Focus the watcher on one service or failure mode and signals that need attention. Only the name, context, and data sources are required. Everything else starts with defaults under Advanced settings, and you can change it later. Successful calibration starts monitoring automatically.

1. In the main menu, expand AI and click Watchers.

2. Click New watcher.

3. Enter a Name.

4. In Give the watcher context, describe what to monitor and expected behavior, for example, brief error spikes during deployments. Include relevant alert rules or dashboards.

5. Select the Datasources that contain its telemetry.

6. Optionally expand Advanced settings to change the defaults. While it’s collapsed, its header shows what the watcher uses: by default, Every 10 minutes, Balanced sensitivity, and no notifications.

   • On the Schedule tab, review Repeats and Sensitivity. Optionally turn on Automatically update checks and choose a Review frequency. Refer to Recalibrate a watcher automatically.
   • On the Notifications and Escalation tabs, add notification or investigation actions. Refer to Manage watcher notifications for setup.
   • On the Labels tab, add labels.

   If a tab has incomplete settings, for example, Slack posting without a channel, the tab shows a warning icon and Create watcher stays unavailable until you complete or turn off the setting.

7. Click Create watcher.

If calibration can’t start, the watcher stays saved. Retry from the form or open the saved watcher to review its status.

Calibrate a watcher

Calibration builds checks and a baseline from your watcher goal and data sources. It starts when you create a watcher, runs in the background, and starts monitoring when it succeeds.

Open Calibration to follow progress. You can leave the page while it runs. If Assistant asks questions, answer them there and click Continue calibration. Only the person who starts calibration can view and answer them. When calibration finishes, review the thresholds, baseline, coverage gaps, and Calibration summary. Calibrated checks are read-only. If they need changes, pause scanning and recalibrate.

Click Preview on a PromQL or LogQL check to inspect fresh data using your own data source permissions. Previews don’t replay saved run evidence and aren’t available for BigQuery, Infinity, or alert checks.

After service changes or repeated noisy findings, click Recalibrate in the page header or Suggest a change on Calibration. Describe what should change in What would you like to change?, for example a new environment to cover, then click Start recalibration. The watcher goal is fixed once its first calibration succeeds, so describe scope changes here. To add or remove a data source, update the data sources on Calibration first. Recalibration pauses scanning, keeps ongoing issues, and starts monitoring when it succeeds. Only the person who starts it can view the requested change and retry it.

If calibration stops, review its explanation, correct the data sources, and click Retry calibration. If the watcher has never been calibrated, you can also correct its goal. Click Cancel setup to keep a new watcher as a draft, or Cancel recalibration to keep the previous checks. Cancelling leaves scanning stopped.

Choose signals to monitor

Watchers support different checks depending on your selected data sources:

• Prometheus and Amazon Managed Service for Prometheus: Monitor metrics with PromQL queries, for example, error rates, latency, or resource usage.
• CloudWatch: Monitor metrics through your Grafana CloudWatch data source. CloudWatch Logs checks aren’t supported.
• Synthetic Monitoring: Monitor results and execution coverage from existing synthetic checks. Configure the Synthetic Monitoring data source with its linked Prometheus data source. Watchers read existing results; they don’t run synthetic tests.
• Loki: Monitor logs with LogQL queries. Watchers look for new error log patterns and keep an inventory of known patterns across runs. They use this inventory to distinguish new patterns from known noise and acknowledged issues, and to detect changes in pattern frequency.
• Tempo: Monitor trace-derived metrics with TraceQL metrics queries, for example, request rates or latency. These checks require Tempo 2.4 or later.
• Pyroscope: Monitor CPU, wall time, allocation, or retained heap profiles for a service. Goroutine, mutex, and block profiles aren’t supported.
• BigQuery: Monitor data with read-only Standard SQL queries. Calibration rejects invalid SQL and queries estimated to scan more than 100 GB.
• Infinity: Monitor JSON or XML from external HTTP APIs. Refer to Check external APIs with Infinity for setup.

Assistant can also include Grafana-managed alert checks when they’re relevant to your watcher goal.

Recalibrate a watcher automatically

Turn on automatic recalibration to keep checks current as your service changes. Assistant reviews the watcher goal, settings, checks, baseline, and recent runs. It validates changes against live data before updating checks, thresholds, or decision guidance. A review can finish successfully without changing anything.

Reviews run in the background without pausing scanning or asking questions. They preserve your goal, data sources, schedule, sensitivity, notifications, lifecycle, and ongoing issues. If you edit the watcher or restore a version during a review, it stops without overriding your change.

To turn on automatic recalibration:

1. Open the watcher and select Configuration.
2. Under Automatic actions, turn on Automatically update checks.
3. Choose a Review frequency. Weekly reviews run on Mondays and Monthly reviews run on the first day of each month, both at 09:00 in your Grafana timezone.

You can also enable this option when you create a watcher. Next review on Configuration shows the next scheduled review. Turn off Automatically update checks to stop future reviews; previous changes stay saved.

To enable reviews, you and the watcher’s creator need grafana-assistant-app.watcher-agents:read, grafana-assistant-app.watcher-agents:write, and grafana-assistant-app.chats:access. Reviews always run as the creator, regardless of who enables them. Turning reviews off only requires grafana-assistant-app.watcher-agents:write.

Reviews run only while the watcher is Active. Reviews due while it’s paused, calibrating, or in setup are skipped, as are reviews when the creator lacks the required permissions. You can’t enable reviews for an archived watcher.

Open Calibration to read the latest Automatic recalibration summary, including changes, unchanged results, or reasons a review stops or is skipped. Click View in Versions to inspect saved changes. To undo them, refer to Review versions and restore a definition.

Run and manage watchers

Open a watcher’s detail page to control scanning and navigate Overview, Ongoing issues, Run history, Calibration, Notifications, Versions, and Configuration. Runs use the stack’s Assistant service account, not your identity.

Use the scanning switch in the page header to pause or resume monitoring. If the watcher is Ready, click Start scanning or Resume scanning on Calibration. Click Run now for an immediate check, including while the watcher is Ready or Paused, without turning on scheduled scanning. The watcher must be calibrated and its automatic stop time must not have passed.

To stop at a chosen time, open Configuration, enable Stop automatically, and set Stop watching at in your Grafana timezone. Select Archive when auto-stop happens if you also want to archive it. A run already in progress can finish.

Click Archive watcher on Configuration and confirm with Archive to stop monitoring and keep configuration and history. To use it again, find it under Archived watchers and click Unarchive, or open Configuration and click Restore watcher. Restoring leaves monitoring paused; clear or extend any past stop time before resuming.

To permanently remove the watcher, click Delete watcher on Configuration, then confirm with Delete. Deleting requires grafana-assistant-app.watcher-agents:delete; editing, calibrating, starting, pausing, archiving, and restoring require grafana-assistant-app.watcher-agents:write.

The Token usage card on Overview estimates tokens per month at the current schedule, or per run for watchers that run only on triggers. The Watchers list shows estimated tokens per hour, monthly usage, the applicable limit, and its reset date. Watcher runs, calibration, and automatic recalibration reviews count toward the same monthly usage under your Grafana Assistant pricing plan. If you reach the limit, runs and calibration stop until reset and scheduled Watchers resume automatically. Investigations consume tokens separately from watcher runs. For details, refer to Understand Grafana Assistant pricing and usage.

Review watcher results

Open a watcher’s Ongoing issues tab to review the issues it’s tracking. Expand an issue’s evidence to inspect the queries and data behind it. You need Watcher read access and access to its data sources to view this saved evidence.

Open Run history and select a run to review its report. Why this verdict explains the assessment, What it’ll watch next lists what the watcher carries into the next run, and Telemetry evidence compares the data with the baseline. For a flagged or escalated run, Telemetry evidence shows the checks behind the verdict first and collapses the remaining checks below them. Each check leads with a one-line summary; click Details to see its query and recorded values.

The Watchers list shows the latest assessment or check status:

For further analysis, click Investigate on an issue or View investigation to open an existing investigation. Click Review to give Assistant feedback to recalibrate the watcher.

Review versions and restore a definition

Open Versions to compare changes or restore an earlier monitoring definition. Calibration and automatic recalibration changes appear as Calibrated versions. Expand a version to read its change notes and compare definitions.

To restore a definition:

1. Expand a version and review its changes.
2. Click Restore on that version’s table row.
3. Review the confirmation and click Restore watcher.
4. Review the restored checks, then turn on the scheduled scanning switch in the page header.

Restoring pauses scanning. It doesn’t restore earlier notification or investigation settings or run state. Wait for calibration or an in-progress run to finish before restoring a version. To undo automatic recalibration changes, restore the preceding version.

Manage watcher notifications

Set destinations when you create a watcher or on Notifications. You can use multiple destinations.

Choose recipients carefully: notifications can include findings and summarized telemetry even when recipients can’t open the watcher in Grafana.

Use Notify for to choose Warnings and critical issues or Critical issues. Recovery notifications for critical issues arrive with either setting.

Understand when Watchers notify

Watchers notify you when an issue needs attention, not on every run. Notifications can report new issues, escalation to critical, reminders for unresolved critical issues, or recovery from critical issues. Warning reminders and recoveries stay silent by default. New destinations don’t receive earlier notifications.

Post to Slack

Enable Post to Slack and choose a channel. If needed, click Open Slack settings to connect the workspace and link your account. Refer to Configure Slack for setup.

If a run doesn’t launch an investigation, click Investigate on its Slack notification, review the Start investigation form, and click Investigate. Otherwise, click Open live investigation to follow the investigation.

To give feedback, click Recalibrate on the notification, describe what should change, and click Recalibrate in the form. Assistant pauses the watcher, recalibrates it with your feedback, and resumes monitoring when calibration succeeds. Follow the link in its thread reply to Calibration to track progress and answer any questions. The button doesn’t require a mention or channel membership for the Grafana app.

You can also reply in the notification thread. In a channel, mention @Grafana and make sure the Grafana app is a member. Assistant confirms the change with you before recalibrating.

Post to Microsoft Teams

If available on your stack, enable Post to Microsoft Teams and enter a channel ID or use Paste channel link. Install Grafana Assistant in the destination team, then click Send test to verify delivery. Refer to Microsoft Teams integration for setup.

Send notifications to a webhook

Enable Send to a webhook, enter a publicly reachable HTTPS URL, and click Send test before saving. The URL can’t include a username or password. Only ports 443 and 80 are supported; IPv6 address literals aren’t.

Click Add authentication to set a Bearer token or HMAC signing secret. Saved secrets aren’t shown again. If you change the host or port, re-enter or clear the credentials. For signed requests, refer to Verify webhook signatures.

Receive webhook deliveries

Your endpoint receives an HTTPS POST with a JSON summary and links to the watcher and any launched investigation. The watcher.url opens the run, and watcher.labels contains the watcher’s labels and is omitted when there aren’t any. The optional investigation.url opens a launched investigation in Workspace. Return a 2xx status within 10 seconds. Redirects aren’t followed.

Use Idempotency-Key or the body’s notificationId to deduplicate retries. Delivery and ordering aren’t guaranteed. Endpoint or credential changes cancel pending deliveries.

Verify webhook signatures

If you set an HMAC signing secret, verify X-Assistant-Signature using the Unix timestamp in X-Assistant-Signature-Timestamp and the raw body:

text Copy

hex(HMAC-SHA256(<SECRET>, <TIMESTAMP> + ":" + <RAW_REQUEST_BODY>))

Replace SECRET with your signing secret, TIMESTAMP with the header value, and RAW_REQUEST_BODY with the unmodified body bytes. Compare the lowercase hexadecimal digest in constant time and reject old timestamps to limit replay attacks.

Send findings to Grafana Alerting

Enable Send to Grafana Alerting, choose a Notification policy if prompted, and set Notify for. Your Alerting policies and contact points control delivery.

Route findings with grafana_alert_source=assistant_watchers and group by watcher_finding_uid. For Grafana IRM, use an IRM contact point. Pausing or archiving resolves published alert instances without necessarily resolving the underlying issue.

Launch investigations

On Configuration, enable Launch on a critical assessment to start Workspace investigations from critical findings. Use Team access to share them with teammates. Without it, access stays limited to you and users with system-created investigation access.

The watcher’s creator needs investigation permissions for launches to succeed.

Review your alerting strategy

Ask Grafana Assistant to review the alert rules and Watchers for a service or scope. Assistant reads both, discusses where an alert, a Watcher, or both fit your telemetry and needs, and can create, update, or recalibrate Watchers for you.

Assistant asks you to approve before it deletes a Watcher or queues an immediate run. When you ask it to recalibrate a Watcher, it passes your feedback to the background calibration. A running Watcher pauses for recalibration and resumes scanning when calibration succeeds. For provisioned alert rules, update the owning source instead, for example, Terraform.

Check external APIs with Infinity

To monitor an external API, select an Infinity data source and ask your administrator to allow the endpoint through its allowedHosts setting. Before the watcher’s first calibration succeeds, include a machine-readable URL in the watcher goal, for example, https://status.example.com/api/v2/summary.json or an RSS feed. To use another endpoint later, ask your administrator to add it, or a URL it starts with, to allowedHosts, then recalibrate.

Infinity supports JSON or XML over GET, without custom headers or a request body. Status endpoints report current state, not historical comparisons. If calibration needs a usable URL, it tells you what to add.

Troubleshoot common issues

Use the run report or a test notification to diagnose a failure.

Investigate failed runs

Open the failed run in Run history and review its error and telemetry evidence. If queries no longer match your service’s labels or data, update the data sources if needed, then recalibrate and describe what changed.

Fix automatic recalibration

Read the Automatic recalibration summary on Calibration for the reason a review stops or is skipped. Resume scanning or restore the creator’s required permissions so the next scheduled review can run. Reviews that stop before finishing try again at the next scheduled time.

If Configuration shows Automatic recalibration setup incomplete, click Retry setup to save the settings for the existing watcher.

Fix webhook delivery

Click Send test on Notifications and review the result. Check that your endpoint has a publicly trusted TLS certificate and accepts requests without redirects. For signature errors, refer to Verify webhook signatures. For missing notifications, refer to Understand when Watchers notify.

Next steps

Configure access and integrations for your watchers.

• Manage Assistant access with RBAC to grant Watchers permissions.
• Grafana Assistant Investigations to learn how critical watcher findings can become Workspace investigations.
• Microsoft Teams integration if you want Watchers to post notifications to Teams.
• Configure Slack if you want Watchers to post notifications to Slack.