Run cost-effective, always-on agents to monitor your environment: the engineering behind Watchers

Alert rules are great for the failures you can predict, but they fall short on the ones you can't. With the advent of modern AI, you might think the solution is to swarm your stack with agents that monitor every last detail, but that can quickly lead to astronomical bills.

And those are the gaps Watchers fill: catching what alerts miss while keeping costs in control. Generally Available as of ObservabilityCON 2026, Watchers are always-on agents in Grafana Assistant. You describe what to monitor (in natural language) and pick your data sources, and the watcher learns what "normal" looks like and tells you when you drift from that state.

Ultimately, this helps you surface problems without needing an alert, giving you wider coverage and more confidence in your system. But how do we keep Watchers cost-effective for our customers, especially in comparison to building your own agent that runn around the clock?

For example, let's say you deploy your own version of a watcher that checks in every 10 minutes, which means it runs 4,320 times a month. If each run were an agentic investigation, which cost us between $0.56 and $1.26 in a recent experiment, that one watcher you built would cost thousands of dollars a month.

Yet the Watchers running on our own Grafana Cloud stack cost a median of less than $5 a month. In this post, I'll show you the engineering behind that number, which mostly comes down to one question: where do you actually need an LLM's intelligence, and how much of it?

The TL;DR on Watchers

Watchers keeps an eye on one things you care about, such as a service, a dependency, or a fix you shipped after an incident. Think of it as the dashboard check you'd otherwise do by hand, done around the clock. It stays quiet until something needs attention, then it sends you its assessment and the evidence behind it. For critical findings, it can also launch an Investigation in Assistant to find the cause.

Here's the gist of how it helps you identify potential issues in a cost-effective manner, which I'll cover in greater detail in the remaining sections in this post:

• The design: A capable agent calibrates each watcher once, so every run after that is plain code plus one call to a small, fast model.
• The cost: Less than $10 a month per watcher for 41 of our 43 internal Watchers. Watchers are free until Dec. 1, 2026, and then will bill as part of Assistant token usage.
• The results: Coverage beyond your alerts, so you catch what you didn’t know to watch for and better trust your system. Our own watcher found more than 10 customer-affecting production bugs in the past few months.
• The integrations: Watchers read most of your telemetry, plus any HTTP API, and send findings to Slack, Microsoft Teams, webhooks, or Grafana Alerting.

Put the intelligence only where it's needed

Keeping an eye on a service comes down to three jobs: 

1. Working out what to watch and what normal looks like
2. Comparing the same signals with normal again and again
3. Deciding whether anything is worth interrupting someone for

Only the first and last jobs need intelligence, and they need it in very different amounts. 

Let's go back to our example from the intro about a watcher that checks in every 10 minutes. Determining what to watch takes exploration and judgment, but once you've made a decision, you don't need to revisit it constantly. After all, the answer barely changes between 9 and 9:10 a.m. And deciding whether something matters enough to interrupt someone takes judgment on every run, but it's a narrow question once the legwork is done. The middle job is fetching data and doing arithmetic, which code does well and LLMs don't. 

So a watcher splits the work three ways:

And here's what that workflow looks like:

Calibration: do the hard thinking once

Calibration is the secret sauce. Think of it as pre-processing: when you create a watcher, a background agent does the setup work an experienced engineer would normally do and writes down what it learns. The expensive reasoning happens once, and every run inherits a narrow, well-prepared task that an LLM is good at.

The calibration agent:

• Settles the scope. It maps where your telemetry lives and asks you up front when it hits a question that only you can answer. For example: "This service runs in 32 production clusters and one ops cluster. Should each cluster be judged on its own?"
• Starts from what your team knows. It mines your alert rules and dashboards first, since they encode failure modes, thresholds, and severities your team already agreed on.
• Measures normal over time. It checks each signal at several points over the past week and reads daily and weekly traffic cycles, so a quiet Sunday night doesn't look like an outage.
• Validates and writes it all down. Every check is tested against live data before it's saved, along with a calibration context: a compact decision guide covering the service, your decisions, deploy behavior, normal ranges, and known noise.

A calibration takes a few minutes at most. Call for call, it's the most expensive thing a watcher does, but it happens once, not every 10 minutes.

Run plain code, then make one model call

A calibrated watcher doesn't explore anything at run time. It executes its saved checks and turns the results into a compact digest, without a model:

• Metrics: Code compares each value with the same time yesterday and with a statistical baseline, then applies the calibrated thresholds, so the model gets verdicts it can't miscompute.
• Logs: Loki's patterns API already groups log lines into patterns at ingestion. The watcher keeps an inventory of them across runs, so it can spot new patterns, notice when a known one doubles in rate, and fetch examples only for what's new.
• Deploys: A check that reports rollouts lets the model tell "this started right after a deploy" from coincidence.

Healthy checks collapse to one line each, and the digest goes to exactly one model call, with no tools. The model returns a structured decision: an assessment (OK, warning, or critical), a short report, the issues it's tracking and their evidence, and whether to notify anyone or launch an investigation.

For example, let's say you roll out a deployment and five minutes later you get an error rate that's slightly up and a log pattern nobody has seen before. Is that noise, or is it worth waking someone up for? This is a task that LLMs are genuinely good at. Weighing evidence like that and explaining the call plays to an LLM's strengths while crunching thousands of data points doesn't. Because the question is this narrow, a small, fast model is enough.

We don't take that on faith, either. Our evals replay real production decisions against candidate models, and the model most watchers use today didn't miss a single critical case, at about half the cost per decision of the model it replaced.

What it costs

Here's what that adds up to for the 43 Watchers running on our internal stack, at our list price of $2 per million tokens:

Even the most expensive one spends less than one cent per run. Cost follows the two things you control: how often a watcher runs (every five minutes to every three hours) and how much it watches. Calibration comes on top, and investigations a watcher launches are billed as investigations. Each watcher's Overview tab estimates its monthly token usage, so you see the cost before your bill does.

Watchers are free until Dec. 1, 2026. After that, they will draw on your plan’s monthly system-initiated allowance (shared with other automated work) and bill on Assistant token usage. See Grafana Assistant pricing for more info.

Note: Watchers will work exactly as they do today when billing goes live, and we expect the cost of running one to remain in line with what we’ve shared here.

What our own watcher found

A cost-effective watcher is only worth it if it catches things. Over the past few months, the watcher our team runs on Grafana Assistant itself found more than 10 production bugs that affected customers, including:

• A crashlooping Microsoft Teams integration
• Broken investigation reports
• Missing webhook credentials in a newly enabled region
• Slack OAuth failures
• Even bugs in Watchers themselves, like alerts that never reached Alertmanager

Many of these surfaced as a new error pattern in the logs. A new pattern alone is often noise, but one that lines up with rising errors and a fresh deploy usually isn't. That's exactly the kind of call a watcher makes well.

It isn't always right about the cause, and it doesn't need to be. During one rollout, it flagged real failures from an exhausted MySQL database but blamed the wrong cause, and engineers found an expensive query plan that likely contributed. The watcher notices; people, or an Assistant Investigation, dig in.

Plug it into your stack

A watcher can sit on top of most of your stack:

• Data sources: Prometheus (including Amazon Managed Service for Prometheus), CloudWatch metrics, Loki, Tempo, Pyroscope, Synthetic Monitoring, Grafana-managed alerts, and BigQuery for business signals. With the Infinity data source, it can also read any HTTP API that returns JSON or XML. One of ours watches the status pages of the LLM providers that Assistant depends on.
• Destinations: Slack, Microsoft Teams, signed webhooks, and Grafana Alerting, so findings flow through your notification policies and contact points, including Grafana Cloud IRM. A critical finding can also launch an Assistant Investigation or trigger an Assistant Automation.

Watchers don't replace alert rules; they cover what you can't spell out in advance. To find the right mix, ask Assistant to review a service, and it recommends an alert, a watcher, or both for each failure mode.

To try it, go to AI > Watchers in Grafana Cloud and create a watcher for a service your team checks by hand after every release. Describe what normal looks like, pick your data sources, and let calibration do the rest. To learn more, refer to the Watchers documentation.

Miss out on ObservabilityCON 2026? Check out our blog post recapping all the big announcements!