Create a watcher
The Watchers page is under AI in the main menu. Create a watcher by describing the service, system, or failure mode to monitor. Keep the scope narrow: one service, namespace, cluster, or failure mode. Include context Assistant can’t infer from telemetry, for example whether short error spikes during deploys are expected.
You don’t need to define what a problem looks like. Calibration works out what normal looks like from your telemetry.
Pick one thing to watch that has live data. Calibration fails if your data sources have no telemetry for what you describe. If you already know a service that’s sending telemetry, skip the next section.
To find a Prometheus job with live data, complete the following steps:
Open Explore.
In the data source picker, choose a Prometheus data source.
If you have more than one, choose the one with your service’s metrics.
Select Code.
In the query editor, next to Metrics browser, enter
count by (job) (up == 1).Click Run query.
Note one job name from the results, for example
integrations/node_exporter.The job name is the value in quotes after
job=. Any job in the list works.
If you ran the query, each job in the results has at least one target that’s up and sending metrics, so you can watch any of them.
To open the create form, complete the following steps:
Open Watchers.
Click New watcher.
You’re on the New watcher form.
To fill in the form and create the watcher, complete the following steps:
In Name, enter the service or check you want to watch.
Pick something your data sources have telemetry for. You’ll look this name up later in the Watchers list.
In Give the watcher context, describe what to watch and anything it can’t learn from telemetry, such as expected rollout behavior, known noise, or acceptable tolerances.
For example: “Watch the payments API in production. Short error spikes during deploys are expected, but errors should recover within a few minutes.”
If you picked a job in Explore, you can write: “Watch the Prometheus job integrations/node_exporter. Raise concern if any host goes down or stops reporting.” Replace the job name with yours.
In Datasources, pick at least one data source, for example Prometheus or Loki, that has telemetry for the service you’re watching.
If you used Explore, pick only the data source you queried there.
Leave Repeats at Every 10 minutes.
You can choose any interval from every 5 minutes to every 3 hours.
Leave Sensitivity at Balanced.
Change it only if you know the watcher should be more or less sensitive.
Click Create watcher.
Leave the rest of the form as it is. For this intro, you don’t need Labels, Stop automatically, the Notifications settings, or Launch on a critical assessment. You can configure notifications later from the watcher’s page.
You created a watcher with a name, context, data sources, cadence, and sensitivity. Grafana opened its Calibration tab, and calibration is now running in the background. In the next milestone, you’ll follow calibration and review the checks it saves.