Documentation Grafana Cloud Platform Assistant Assistant capabilities Assistant Watchers

Use Assistant Watchers

Assistant Watchers are always-on agents that check a scoped part of your telemetry on a schedule. You describe what the watcher should monitor, Assistant calibrates concrete checks against your data, and the watcher raises concern when the latest run finds something notable.

Use Watchers for recurring service health checks, noisy-but-important failure modes, and follow-up monitoring after an incident. A watcher can stay quiet when signals look normal, notify Slack, Microsoft Teams, or a webhook endpoint, and launch an Assistant Investigation for critical findings.

Before you begin

Confirm that your stack and user account can create and run Watchers before you start. Watchers run in the background, so setup affects both the authoring flow and future scheduled runs.

• Grafana Cloud: Watchers are a Grafana Cloud-only public preview feature.
• Grafana Assistant: Assistant must be enabled for your stack, and any required Assistant terms must be accepted.
• Data sources: Select at least one Prometheus, Amazon Managed Service for Prometheus, Loki, or BigQuery data source for calibration. Assistant can also include Grafana-managed alert checks when they’re relevant. To read an external HTTP API, such as a provider’s status page, also select an Infinity data source whose allowedHosts includes that API.
• Data access: When you create or edit a watcher, you can only add data sources you’re allowed to query, and alert checks require permission to read alert instances. Watcher runs then query them as the stack’s Assistant service account, so a watcher keeps running when its creator leaves the organization.
• Label-based access control (LBAC): Watchers don’t support data sources that have team LBAC rules, because team LBAC rules don’t restrict the Assistant service account. You can’t add such a data source to a watcher, and an existing check against one fails.
• Permissions: You need plugins.app:access scoped to plugins:id:grafana-assistant-app and the relevant grafana-assistant-app.watcher-agents:* permissions: :read to view Watchers, :create to create a watcher and run it immediately, and :write to calibrate, start, and pause it. Refer to Manage Assistant access with RBAC for details.
• Slack notifications (optional): To post watcher results to Slack, connect Slack for the stack and link your Slack account.
• Microsoft Teams notifications (optional): If Microsoft Teams integration is enabled for your stack, connect Microsoft Teams for the stack and install Grafana Assistant in the destination team.
• Webhook notifications (optional): To send watcher notifications to your own systems, you need an HTTPS endpoint that’s reachable from the public internet and accepts POST requests with a JSON body.
• Slack thread recalibration (optional): To recalibrate Watchers from Slack notification threads, the Grafana app must receive replies in that thread. For channel notifications, add the Grafana app to the channel and mention @Grafana in the reply.
• Investigations (optional): To launch investigations from critical watcher findings, you need Assistant investigation access.

Open Assistant Watchers

Open the Watchers page from the AI section of the main menu to view the watcher fleet, create a watcher, and review recent results. The page shows each watcher’s lifecycle status, latest assessment, schedule, and recent activity.

On wider screens, switch between Cards and Table to scan watchers in the format you prefer. The table shows recent run history when runs are available. On narrower screens, Watchers uses the card view.

1. In the main menu, expand AI.
2. Click Watchers.
3. Review the list of existing watchers, or click New watcher to open the watcher form.

The Watchers list uses these assessment labels:

Review your alerting strategy

Use Alerts & Watchers mode in Assistant to review alerting for a service, team, folder, or namespace. Assistant inventories current alert rules and Watchers, checks firing and notification history when it is available, and organizes findings around the failure modes that matter to the scope.

For each failure mode, Assistant recommends an alert, a Watcher, both, dashboard-only coverage, or no notification. The recommendation depends on service maturity, telemetry quality, urgency, operating preferences, and maintenance appetite. For example, a new service can benefit from a flexible, lower-maintenance Watcher while its normal ranges are still changing. A critical service can use a deterministic alert for dependable paging and a Watcher as a second detection and contextual-triage layer.

Assistant explains the evidence and trade-offs, then asks which approach you prefer before changing a Watcher. To change what a Watcher checks, Assistant updates the Watcher goal with your decisions, after you agree to the new wording, and starts calibration in the background. It doesn’t edit checks or thresholds itself. For Grafana-managed alert rules, Assistant can also apply the proposed rule change after you review and approve it. If a rule is managed in Terraform, OpenTofu, Pulumi, or another provisioning source, Assistant tells you to update the owning source instead. If history is unavailable, Assistant explains which evidence is missing.

To start a strategy review:

1. Open Grafana Assistant.
2. Switch to Alerts & Watchers mode.
3. Describe the service, team, namespace, or folder you want to review.
4. Review the recommended split between alerts, Watchers, hybrid coverage, dashboard-only coverage, or no notification.
5. Choose the approach you want before you create or recalibrate a Watcher.

Create a watcher

Create a watcher by filling the watcher form. After you review the final configuration, click Create watcher to create the Watcher and start its calibration in the background.

1. In the main menu, expand AI and click Watchers.

2. Click New watcher.

3. Enter a Name that makes the watcher easy to recognize later.

4. In Give the watcher context, describe what the watcher should monitor. Include important service names, expected rollout behavior, known noise, and acceptable tolerances.

5. Select one or more Datasources. Pick the Prometheus, Amazon Managed Service for Prometheus, Loki, and BigQuery data sources that contain the relevant metrics, logs, or analytical data. Add an Infinity data source only when the watcher should read an external HTTP API.

6. Choose how often the watcher Repeats. The default is Every 10 minutes. The interval must be between 5 minutes and 3 hours. Shorter intervals detect changes sooner but run more often, so they consume more tokens.

7. Choose Sensitivity to control how readily the watcher raises concern. The default is Balanced. Keep it unless you know the watcher should be more or less sensitive.

8. Optionally add Labels when you want consistent search, filtering, or downstream routing context on every published finding and notification. The label editor suggests keys and values from alert rules you can access, existing watchers, and Grafana Labels. You can also enter custom values.

9. Configure Slack, Microsoft Teams, webhook, or investigation actions if you want the watcher to notify teammates or escalate findings automatically.

10. Click Create watcher.

    Clicking Create watcher creates the watcher, starts calibration in the background, and opens the watcher’s Calibration tab. Filling the form doesn’t save a watcher. If calibration can’t start, the watcher still exists. Click Create watcher again to retry.

Use context that Assistant can’t infer from telemetry alone. For example, tell the watcher whether short error spikes during deployment are expected, which dependencies are noisy, and which symptoms always require attention.

Calibrate a watcher

Calibration turns your monitoring intent into durable checks that future unattended runs can evaluate. Assistant calibrates each watcher in the background: it inspects relevant infrastructure, dashboards, alert rules, and live telemetry, then saves validated queries and a calibration baseline. When calibration succeeds, the watcher starts scanning on its schedule.

Follow calibration on the watcher’s Calibration tab, which shows each step while it runs. Calibration continues when you leave the page. If your browser allows notifications from Grafana, it notifies you when calibration finishes, fails, or needs your input.

Early in calibration, Assistant can stop to ask a few questions about scope or intent, for example which clusters or environments to cover. Answer them on the Calibration tab and click Continue calibration. Only the person who started calibration can answer. Calibration doesn’t ask you for queries or thresholds, because it derives them from your telemetry.

Calibration can create PromQL, LogQL, BigQuery SQL, and alert checks. BigQuery checks use read-only Standard SQL and run a dry-run validation before they’re saved. The dry run rejects invalid SQL and queries estimated to scan more than 100 GB. For Grafana-managed alert checks, calibration tests the Alertmanager label matchers before saving the check. If the alert rule exists only in a datasource-native Alertmanager, calibration uses the alert rule’s exact PromQL expression instead. Calibration records normal ranges, warning and critical thresholds, known benign patterns, coverage gaps, and any current conditions that should be treated as already acknowledged.

If calibration can’t establish a complete baseline, for example because the selected data sources don’t contain the telemetry the goal describes, it stops and explains what’s missing. Correct the watcher goal or data sources on the Calibration tab and click Retry calibration, or click Cancel setup to return the watcher to draft.

Recalibrate a watcher

Recalibrate a watcher when its checks no longer match the monitored system, for example after a service change or repeated noisy findings. Recalibration revises the existing checks and baseline instead of starting over, and it keeps the watcher’s ongoing issues.

Calibration works from the watcher goal and data sources, so first update the goal with what should change, such as a new environment to cover or a condition that’s expected during deployments, and adjust the data sources if needed. Then click Recalibrate. A watcher that’s scanning pauses while it recalibrates and resumes scanning when recalibration succeeds. Click Cancel recalibration to stop and keep the previous checks.

Check external APIs with Infinity

A watcher can read JSON or XML from an external HTTP endpoint through an Infinity data source, for example the status pages of the providers a service depends on. Infinity checks are opt-in: an administrator controls which origins are reachable with the Infinity data source’s allowedHosts setting, and calibration adds Infinity checks only when you select that data source for the watcher.

Include the exact machine-readable URLs in the watcher context, for example https://status.example.com/api/v2/summary.json or a service’s RSS feed. Calibration requests and saves only URLs written exactly in the watcher context, already saved as checks, or listed as full URLs in the Infinity data source’s allowedHosts. Administrators can list a full URL there, such as https://status.example.com/api/v2/summary.json, so that watcher owners don’t need to know it. It doesn’t guess URLs or follow links from responses, even on an allowed origin. If a source you mention has no usable URL, calibration records the gap and tells you which URL to add.

Each run sends one GET request per check, without custom headers or a request body, and never asks for approval. A check can either set thresholds on numeric fields or pull data, such as the latest incident title, that the watcher reads on every run. Checks that pull data without thresholds send every run to the decision model. Status endpoints return only their current state, so these checks have no 24-hour comparison.

After calibration finishes, the watcher starts scanning on its schedule. Review the generated checks and baseline on the Calibration tab, and pause scheduled scanning if you want to change them before the next run.

In the Calibration tab’s query list, click Preview on a PromQL or LogQL check to inspect live data from its data source. A preview requires a query expression and a selected data source; other check types don’t offer a live preview. The preview uses a recent time range based on the watcher’s run interval.

Start, pause, and run a watcher

Use lifecycle controls to decide when a watcher actively scans telemetry. A watcher can be ready but inactive, active on a schedule, paused, or run once on demand.

• A watcher starts scanning when its calibration succeeds. If a calibrated watcher is Ready but not scanning, for example after you cancel a recalibration, click Start scanning on the calibration banner of the Calibration tab to turn on scheduled runs. After a watcher has completed a run, this control reads Resume scanning.
• Use the scheduled scanning switch in the rail to the right of the tabs to pause scheduled runs without deleting the watcher. The label beside the switch states the current lifecycle: Ready before you start scanning, Active while scanning is on, and Paused after you pause.
• Click Run now to trigger one immediate run outside the schedule. Run now also works while the watcher is Ready, so you can inspect a result before you start scanning.
• Click Archive to stop scanning while keeping the watcher’s configuration and run history. Find it under Archived watchers in the list, then click Unarchive to restore it. Restored watchers remain paused until you start scanning again. Restoring requires an available watcher slot.
• On the Configuration tab, click Delete watcher, then confirm with Delete, to remove the watcher and stop future runs.

Scheduled runs do not overlap for the same watcher. If a run is already in progress, the next run waits or is skipped so the watcher doesn’t scan the same scope concurrently.

Stop a watcher automatically

For a rollout or another limited monitoring window, enable Stop automatically when you create the watcher or open Configuration on its detail page. Set Stop watching at in your Grafana timezone. You can also ask Assistant to set the stop time in Alerts & Watchers mode.

At the stop time, new scans stop and the watcher pauses. A scan already in progress can finish. Configuration and history remain available. Select Archive when auto-stop happens to archive the watcher when the stop is processed. You can change or clear the stop time before it happens; clearing it also cancels automatic archiving.

To use the watcher again, restore it if it’s archived, extend or remove the stop time, then turn on scheduled scanning. Watchers have no stop time unless you set one.

Run a watcher when a pull request merges

To check a service right after a change ships, have a running watcher also run when a pull request merges. By default this adds runs to the schedule, and you can also make the watcher run only on merges.

Before you begin, connect GitHub and select the repositories in GitHub settings. Only selected repositories can trigger runs. You also need to link your GitHub account, and you can add triggers only for repositories your GitHub account can access. If your account isn’t linked, Add trigger offers to connect it.

1. Open Configuration on the watcher detail page.
2. Under When to run, click Add trigger in Also runs when.
3. Choose a repository.

To stop following a repository, click the remove icon next to its trigger. You can remove or change a trigger that someone else added, even for a repository your GitHub account can’t access.

To give a deploy time to roll out, choose a delay for the trigger, for example Run 15 minutes later. The watcher then runs that long after the merge.

To run the watcher only on merges, set Repeats to Only when triggered. The watcher then has no schedule, and Looks back sets how much recent telemetry each run checks, ending when it runs. A watcher that runs only on triggers keeps at least one trigger. To remove its last trigger, choose a Repeats interval first.

When a pull request merges into a chosen repository’s default branch, the watcher runs once. Merges while the watcher is paused, calibrating, or archived don’t start a run, and a merge that arrives during a run is skipped. The run appears in the history with on merge, and its details link to the pull request. Only people whose GitHub account can access the repository see which pull request started the run. The watcher considers the pull request as a possible cause when telemetry changes after the merge.

If you remove a repository from GitHub settings, its merges stop running the watcher. If you rename a repository on GitHub, select its new name in GitHub settings so its merges keep running the watcher.

Review watcher results

Review each watcher from its detail page to understand what it checked, what it found, and what state it carried forward.

The Overview tab shows ongoing issues and Run history. Use Run history to compare assessments over time.

Standing state, in the rail to the right of the tabs, shows the latest assessment. The rail also carries the scheduled scanning switch, Run now, cadence, last run, next run when scanning is on, Token consumption, and who created the watcher.

The Notifications tab shows the current delivery and escalation configuration. The Versions tab keeps the latest 100 finalized definitions for comparison and restore.

The Calibration tab lists the watcher goal, its Datasources, Calibration baseline, and the calibrated checks, including enabled and disabled queries. To add or remove a data source, change the Datasources selection and click Save datasources. Saved checks keep their data sources, so recalibrate the watcher to use the new selection. You can’t change data sources while a calibration is running.

Open a run in Run history to inspect the report. Why this verdict explains why the watcher chose All clear, Flagged, or Escalated. Telemetry evidence shows the compact current-versus-baseline data that informed the decision.

For an ongoing issue, expand its linked query evidence to inspect the query, recorded values, and a data preview for the run’s telemetry window. Older runs without a recorded window show the recorded values without a historical chart.

When an ongoing issue stays open, you can continue it in a Workspace investigation or hand it back to Assistant to review your feedback. Assistant can then update the watcher goal with your feedback and recalibrate the watcher.

Review versions and restore a definition

Use Versions on the watcher detail page when you want to compare the current monitoring definition with earlier finalized versions or restore a known-good configuration.

Each retained version shows a summary of what changed. Expand a version to review the diff and raw JSON for that definition. The page keeps the latest 100 versions, so the earliest retained version might not have an older definition to compare.

To restore a definition:

1. Open the watcher and select Versions.
2. Expand the version you want and review its changes.
3. Click Restore for that version.
4. Review the confirmation and click Restore watcher.
5. Review the restored definition, then turn on Resume scheduled scanning when you want scheduled scans to resume.

When you restore a version, Assistant pauses the current watcher first and creates a new version from the restored definition. It doesn’t restart scheduled scanning automatically, and it doesn’t copy lifecycle state, actions, or learned run state from the earlier version. Wait for calibration to finish before you restore a version.

Configure notifications and investigations

Configure actions when you want Watchers to notify teammates or start deeper analysis automatically. Actions are evaluated after a run completes and the watcher decides whether the concern is worth attention.

Each run makes one notification decision. Slack, Microsoft Teams, and webhook destinations can be enabled together, individually, or not at all, and each destination applies its own Notify for severity filter to that shared decision. Configure notifications when you create the watcher or later from the watcher detail page. Investigation launch is a separate action for critical findings.

For each destination, choose Warnings and critical issues (the default) or Critical issues. Critical issue recovery updates bypass this filter, even for a destination set to Critical issues.

Understand when Watchers notify

Watchers suppress repeat notifications so a persistent issue doesn’t page you on every run. A run sends a notification only when the watcher judges the situation worth reporting and at least one of these changes occurred:

• A new issue appears.
• An issue last notified as a warning escalates to critical.
• A previously notified critical issue is still open and enough time has passed since the last notification, about four hours by default.
• A previously notified critical issue genuinely resolves.

Warning reminders and warning resolutions are silent by default. A run with no eligible change stays quiet, even when previously reported issues remain open. Enabling a new destination doesn’t replay earlier notifications; the destination receives notifications starting from the next eligible run.

Post to Slack

To post to Slack, enable Post to Slack and choose a Slack channel. If Slack isn’t connected, open Slack settings from the alert shown in the form, connect the workspace, and link your account.

When a Slack notification makes the wrong call or needs better context, reply in the notification thread to recalibrate the watcher. In a direct message, reply in the thread. In a channel, reply in the thread and mention @Grafana; the Grafana app must be a member of that channel.

Post to Microsoft Teams

If Microsoft Teams integration is enabled for your stack, you can post notable watcher assessments to a Teams channel.

Enable Post to Microsoft Teams, then enter the channel ID or click Paste channel link and paste the link copied from More channel options > Copy link in Teams. The destination team must already have the Grafana Assistant app installed. After you select the channel, click Send test to confirm delivery before you click Create watcher or save your changes.

If Teams isn’t connected for the stack, click Open Teams settings from the Watcher form. For setup details, refer to Microsoft Teams integration.

Send notifications to a webhook

To send watcher notifications to your own tooling, for example, an incident management system, an event bus, or an internal service, enable Send to a webhook and enter the endpoint URL. Optionally, click Add authentication to configure:

• Bearer token: sent with every delivery in the Authorization header as Bearer <TOKEN>.
• HMAC signing secret: signs every delivery so your endpoint can verify that the request came from Grafana Assistant. Refer to Verify webhook signatures.

The endpoint URL must use HTTPS, resolve to a publicly reachable host, and must not include a username or password. Only ports 443 and 80 are accepted, and IPv6 address literals aren’t supported. Assistant treats the URL, bearer token, and signing secret as secrets: they’re stored encrypted, they’re never returned by the API or the UI after saving, and only a sanitized host preview is shown. To change a saved secret, enter a new value or reset it; you can’t read the existing one back.

Changing Notify for applies to future notification decisions. Unlike an endpoint or credential edit, it doesn’t cancel deliveries that are already queued.

Receive webhook deliveries

Assistant delivers each notification as an HTTPS POST request with a JSON body. Respond with any 2xx status code to acknowledge the delivery. Each request includes these headers:

The body is versioned and contains an operational summary. It has no dedicated fields for query expressions, log lines, or full telemetry digests, though notification.message can include summarized telemetry values. The following example shows a critical notification from a watcher that also launched an investigation:

JSON Copy

{
  "version": "1",
  "type": "watcher.notification",
  "notificationId": "0b9e4a83-6f2d-5c3e-9d41-a1b2c3d4e5f6",
  "timestamp": "2026-08-04T14:30:00Z",
  "watcher": {
    "id": "4f8c1a2b-3d4e-5f60-7a8b-9c0d1e2f3a4b",
    "name": "Checkout health",
    "url": "https://example.grafana.net/a/grafana-assistant-app/watchers/4f8c1a2b-3d4e-5f60-7a8b-9c0d1e2f3a4b"
  },
  "run": {
    "id": "7d6e5f4a-3b2c-1d0e-9f8a-7b6c5d4e3f2a",
    "assessment": "critical"
  },
  "notification": {
    "assessment": "critical",
    "message": "Checkout errors are elevated above the calibrated baseline.",
    "ongoingCritical": 1,
    "ongoingWarning": 0
  },
  "investigation": {
    "id": "2a1b0c9d-8e7f-6a5b-4c3d-2e1f0a9b8c7d",
    "url": "https://example.grafana.net/a/grafana-assistant-app/investigations/2a1b0c9d-8e7f-6a5b-4c3d-2e1f0a9b8c7d"
  }
}

The notification.assessment field reflects the severity of what changed in this run: critical for critical findings, warning for warning-level findings, and ok when the notification announces a resolved critical issue. The investigation object is present only when the run launched an investigation. Payload version 1 evolves by adding fields; treat unknown fields as forward-compatible additions.

Expect these delivery characteristics:

• A notification can be delivered more than once, or not at all. A retry can arrive after your endpoint already processed the request, so deduplicate by notificationId; a notification that fails all delivery attempts is missed rather than queued indefinitely. Redirects aren’t followed.
• The body and idempotency key are identical across retries. The signature timestamp can change between attempts.
• Notifications aren’t guaranteed to arrive in the order their runs completed.
• Changing the webhook configuration cancels deliveries that are still pending, so a notification accepted for one endpoint or credential is never sent to a newly configured one.

Verify webhook signatures

When you configure an HMAC signing secret, each webhook request includes the Unix timestamp in X-Assistant-Signature-Timestamp and a lowercase hexadecimal HMAC-SHA256 digest in X-Assistant-Signature. Grafana Assistant calculates the digest as:

text Copy

hex(HMAC-SHA256(<SECRET>, <TIMESTAMP> + ":" + <RAW_REQUEST_BODY>))

Use the request body bytes exactly as received, compare the digest in constant time, and reject timestamps outside an acceptable window to limit replay attacks.

Launch investigations

To launch an investigation, enable Launch on a critical assessment. When the watcher escalates a critical finding, Assistant opens a Workspace investigation seeded with the watcher’s findings. Add Team access when teammates need to view launched investigations. If you leave team access empty, investigations stay visible only to you and users with sufficient system-created investigation access.

Understand how Watchers use data

Watchers use the saved calibration output, not the calibration activity, when they run later. This keeps scheduled runs bounded and easier to audit.

Each run uses:

• The watcher’s monitoring prompt.
• The calibrated PromQL, LogQL, BigQuery SQL, Infinity, and alert checks.
• The calibration baseline and thresholds.
• Recent ongoing issues and short-term notes.
• Current telemetry returned by the enabled checks.

Watchers run their checks and decide each result as the stack’s Assistant service account, not as the creator. Investigations that a watcher launches run with the creator’s permissions. If the creator can no longer run Watchers, the run still completes and notifies, and reports that the investigation couldn’t launch. Data sources that have team LBAC rules can’t be added to a watcher, and checks against them fail, so use data sources without team LBAC rules for Watchers.

There isn’t a limit on the number of Watchers you can create or restore. Scheduled runs still share the monthly Watcher token limit, so adding Watchers doesn’t increase your available usage.

Watcher runs consume Assistant tokens, but that usage isn’t billed during public preview. The watcher detail page shows estimated token consumption per run and per hour based on recent completed runs. It also shows the active monthly usage meter for Watchers, including the scope of the limit and the reset date. The meter caps how much Watchers can run rather than charging you for those runs. If the monthly limit is reached, watcher runs are blocked until the reset date. Scheduled Watchers remain active and resume automatically after the limit resets. Tokens used by investigations launched from Watchers are separate from the watcher run estimate.

Apply best practices

Follow these guidelines to make Watchers useful and reduce noisy notifications.

• Start with a narrow, specific scope, such as one service, namespace, cluster, or failure mode.
• Calibrate against existing alert rules and dashboards where possible, because they often encode trusted thresholds and service ownership.
• Keep checks focused on symptoms that require attention, not every available signal.
• Review generated thresholds and coverage gaps after calibration, and pause scheduled scanning if they need changes.
• Recalibrate after major service changes, dashboard changes, alert threshold changes, or repeated noisy findings.
• Use Slack actions for findings that need human attention, webhook actions to feed notifications into your own tooling, and investigation actions for critical findings that need deeper analysis.

Troubleshoot common issues

Use these checks when a watcher doesn’t behave as expected. Most issues come from missing permissions, missing data source access, or calibration that no longer matches the monitored system.

Restore Watchers visibility

Confirm that Grafana Assistant is enabled for your Grafana Cloud stack and that you have grafana-assistant-app.watcher-agents:read plus plugins.app:access scoped to plugins:id:grafana-assistant-app. Watchers isn’t available in self-managed Grafana deployments.

Fix watcher creation

Confirm that you have grafana-assistant-app.watcher-agents:create, at least one selected data source, and access to every selected data source. If the create form shows Slack setup warnings, either complete Slack setup or disable Slack posting before you click Create watcher.

Finish calibration

Open the watcher’s Calibration tab and review the explanation of why calibration stopped, for example unanswered questions, unsupported data sources, or telemetry that returned no data. Answer any questions, or correct the watcher goal or data sources and click Retry calibration. Calibration must save at least one enabled, validated query before a watcher can start. Calibrating requires grafana-assistant-app.watcher-agents:write.

If calibration reports that Grafana-managed Alertmanager is unavailable for an alert check, confirm that it’s reachable and that the alert rule’s labels match what the watcher goal describes. When the rule exists only in a datasource-native Alertmanager, calibration uses the rule’s exact PromQL expression instead of Alertmanager label matchers.

Resume scheduled runs

Confirm that you have grafana-assistant-app.watcher-agents:write, and that the watcher is Active and has a valid interval. If the watcher is Ready, click Start scanning on the calibration banner, or Resume scanning if the watcher has completed a run. If it’s Paused, turn on the scheduled scanning switch in the rail to resume scheduled runs.

Investigate failed runs

Open the failed run from the watcher detail page and review the error and telemetry evidence. Check whether a check uses a data source that has team LBAC rules, whether a query references labels that no longer exist, and whether the watcher needs recalibration.

If the run failed because the monthly usage limit was reached, wait until the reset date shown on the watcher page. Scheduled Watchers resume automatically after the reset.

Fix Slack delivery

Confirm Slack is connected for the stack, the creator linked their Slack account, and the selected channel is valid. If channels can’t be loaded, enter the channel ID manually or reopen Slack settings.

Fix Microsoft Teams delivery

Confirm Microsoft Teams is connected for the stack, the Grafana Assistant app is installed in the destination team, and the saved channel value is a channel ID rather than a team or message link. If you paste a link, copy it from the channel’s More channel options > Copy link menu, then use Send test to verify delivery before you click Create watcher or save your changes.

Fix webhook delivery

Confirm the webhook is enabled and the endpoint responds with a 2xx status within 10 seconds, directly at the configured URL, over HTTPS with a publicly trusted TLS certificate. Assistant retries timeouts, connection failures, and 408 or 5xx responses for up to three total attempts; redirects, TLS validation failures, and 4xx responses other than 408 fail immediately.

Configuration edits cancel deliveries that were pending during the edit; new notifications use the updated configuration. If your endpoint rejects deliveries during signature verification, confirm it computes the digest over the raw request body bytes and the timestamp from X-Assistant-Signature-Timestamp, and update it after you reset a secret in Grafana. If a notification seems missing, check whether the run made an eligible change; refer to Understand when Watchers notify.

Next steps

• Manage Assistant access with RBAC to grant Watchers permissions.
• Grafana Assistant Investigations to learn how critical watcher findings can become Workspace investigations.
• Microsoft Teams integration if you want Watchers to post notifications to Teams.
• Configure Slack if you want Watchers to post notifications to Slack.