--- title: "Run private probes in a cluster | Grafana Cloud documentation" description: "Run several Synthetic Monitoring agents as a cluster that shares one private probe and splits its checks, on Kubernetes, Helm, Docker, or Linux packages." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Run private probes in a cluster > Caution > > Clustering is an [experimental](https://grafana.com/docs/release-life-cycle/) feature. Configuration flags and behavior might change in future releases. Clustering lets several agents share one private probe and split the probe’s checks between them. The agents form the cluster with a gossip protocol, and the cluster is eventually consistent. To learn how this affects your checks, refer to [Cluster states and lifecycle](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/clustering/#cluster-states-and-lifecycle). A non-clustered agent behaves the same as a single-node cluster. You configure clustering with `-cluster-*` command-line flags. Before you set up a cluster, [set up a private probe](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/) and install or deploy the agent. To decide whether you need a cluster, refer to [Choose a deployment model](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#choose-a-deployment-model). ## How checks are distributed Every agent in the cluster connects to Synthetic Monitoring with the same probe token and receives the full set of checks assigned to the probe. Each agent then runs only the checks that it owns. The agents use consistent hashing to decide which agent owns each check. Each agent calculates ownership locally from the current cluster membership, so check ownership is never sent over the network. When an agent joins or leaves the cluster, the agents recalculate ownership. Only about 1/N of the checks move, where N is the number of agents, which keeps disruption small. When an agent leaves or fails, the remaining agents pick up its checks. Every agent reports its results under the same probe, so the probe looks the same in Synthetic Monitoring whether it runs on one agent or many. > Note > > Clustering spreads checks across agents. It doesn’t run the same check on several agents for redundancy. Each check runs on one agent at a time. ## Cluster requirements Every agent in the cluster must: - Use the same probe authentication token. - Run the same version of the agent. - Have the same [feature flags](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/configure/#configure-feature-flags) enabled. - Use the same image variant, for example the `-browser` image. - Use the same [browser pool](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/browser-pool/) configuration. - Use the same `-cluster-listen-port` and the same `-cluster-name`. - Accept TCP connections from the other agents on the cluster listen port, `7946` by default. Refer to [Cluster traffic](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/network-and-resource-requirements/#cluster-traffic). - Have a unique node name. The node name defaults to the hostname. Check ownership doesn’t take an agent’s capabilities into account. If an agent that can’t run a scripted or browser check owns that check, the check doesn’t run on any agent. On Kubernetes, deploy the agents as a Deployment with a headless Service. The headless Service lets agents discover each other. ## Best practices ### Plan for heavy checks The cluster balances the *number* of checks across agents, not their cost. Check types use very different amounts of resources: a browser check uses far more CPU and memory than an HTTP check. If a probe runs a few heavy checks among many light ones, some agents can end up with more load than others. In this case, give each agent more CPU and memory, or move the heavy checks to a separate probe. Browser checks use the most resources, so they’re the most likely to make some agents busier than others. If the probe runs more than a few browser checks, run them on a [browser pool](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/browser-pool/). The agents then do similar amounts of work for each check, so load stays more even across the cluster. ### Size the cluster for peak load Checks with long frequencies, such as hourly browser checks, run in bursts. Autoscalers react too slowly to these bursts, and scaling down moves checks to other agents, which then run those checks again right away. For probes that run heavy checks at long frequencies, run a fixed number of agents that can handle the peak load instead of autoscaling aggressively. For sizing guidance, refer to [Size clustered agents and browser pools](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/network-and-resource-requirements/#size-clustered-agents-and-browser-pools). ### Use `-cluster-wait-for-size` with care When a new agent starts, it waits for the cluster to reach `-cluster-wait-for-size` agents before it runs any checks, so it doesn’t briefly run checks that other agents already own. If the cluster doesn’t reach that size within `-cluster-wait-timeout`, the agent runs checks anyway with the agents that are available. Set `-cluster-wait-for-size` well below your usual number of agents, for example below your HorizontalPodAutoscaler or PodDisruptionBudget minimum. Keep `-cluster-wait-timeout` set. If it’s `0`, an agent can wait forever and never run checks. ### Only enable clustering when you need it If a single agent can handle all the checks assigned to the probe, run a single agent. ## Configure clustering on Kubernetes These steps extend [Deployment with Kubernetes](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#deployment-with-kubernetes). They reuse the `synthetic-monitoring` namespace and the `sm-agent-1` secret from that section. Instead of `deployment.yaml`, they use the cluster files from the [Kubernetes example](https://github.com/grafana/synthetic-monitoring-agent/tree/main/examples/kubernetes) in the agent GitHub repository. 1. Get the cluster deployment files. Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/cluster-service.yaml wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/cluster-deployment.yaml ``` `cluster-service.yaml` defines a headless Service so agents can discover each other. `cluster-deployment.yaml` defines a Deployment that runs three agents. In these files: - `--cluster-join-addresses` points to the headless Service. The name resolves to the IP addresses of the agent Pods. - `publishNotReadyAddresses: true` includes Pods that aren’t Ready yet in the Service’s DNS records. If you add a readiness probe to the agents, this setting lets starting agents find peers that haven’t passed the probe yet, instead of each agent starting as a single-node cluster. - The node name defaults to the Pod hostname. - `--cluster-wait-for-size=2` is below the number of replicas, so agents can still start when one Pod is unavailable. - `terminationGracePeriodSeconds` leaves time for an agent to hand its checks over to the other agents before it stops. Keep it above 5 seconds. 2. If you deployed a single agent with `deployment.yaml`, delete it first. The API rejects the clustered agents while the single agent is connected. The probe is offline until the cluster starts. Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl delete deployment sm-agent-1 -n synthetic-monitoring ``` 3. Deploy the headless Service and the agents. Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply -f cluster-service.yaml kubectl apply -f cluster-deployment.yaml ``` 4. Verify that the agents formed a cluster. Each agent logs the cluster members when it joins and whenever membership changes. List the agent Pods, then check the logs of one of them: Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl get pods -n synthetic-monitoring kubectl logs -n synthetic-monitoring POD_NAME | grep "cluster peers changed" ``` Replace `POD_NAME` with the name of one of the agent Pods, for example `sm-agent-7c9d8b5f4-x2kqp`. You can also check that the `sm_agent_cluster_size` metric on each agent equals the number of replicas. Refer to [Monitor the cluster](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/clustering/#monitor-the-cluster). To change the number of agents, scale the Deployment. The agents redistribute checks automatically. ## Configure clustering with Helm If you deploy the agent with the [Helm chart](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#deploy-the-agent-with-helm), enable clustering in your values file: YAML ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```yaml replicas: 3 cluster: enabled: true # Keep it below replicas, so agents can still start when one Pod is unavailable. waitForSize: 2 ``` For a new release, install the chart with this file. For an existing release, upgrade it: Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm upgrade sm-agent oci://ghcr.io/grafana/helm-charts/synthetic-monitoring-agent \ -n synthetic-monitoring -f values.yaml ``` > Warning > > To switch a running single agent to a cluster, first stop the single agent, for example by upgrading the release with `replicas: 0`, then upgrade with the cluster values. A direct upgrade never completes: the API rejects the clustered agents while the single agent is connected. The probe is offline between the two upgrades. The chart sets the join addresses to a headless Service, the advertise address to the Pod IP, and the cluster name to a name derived from the release. It also creates a PodDisruptionBudget, so that node drains evict one agent at a time. Optionally, the chart can also create a NetworkPolicy that only allows cluster traffic from the agents of the release. Refer to the chart [README](https://github.com/grafana/synthetic-monitoring-agent/tree/main/charts/synthetic-monitoring-agent). The chart can also add a HorizontalPodAutoscaler for the agents with `autoscaling.enabled`. Every scale event moves checks between agents, so size the minimum number of agents for peak load. If the probe runs more than a few browser checks, use a [browser pool](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/browser-pool/), so that agent load stays even. Refer to the chart [README](https://github.com/grafana/synthetic-monitoring-agent/tree/main/charts/synthetic-monitoring-agent) before you enable autoscaling. ## Configure clustering with Docker Pass `-cluster-enabled` and `-cluster-join-addresses` to each agent. The first agent that starts a new cluster can leave out `-cluster-join-addresses`. Each agent advertises an address that the other agents use to reach it. By default, the agent uses the first address it finds on the `eth0` or `en0` interface. If the other agents can’t reach that address, for example when agents run on different hosts, set `-cluster-advertise-address` explicitly. The following example runs three agents on the same host. It uses a user-defined Docker network so the containers can reach each other by name. Before you run it, export `SM_AGENT_API_TOKEN` and `API_SERVER` as described in [Deploy the agent using Docker](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#deploy-the-agent-using-docker). Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash docker network create sm-cluster for i in 1 2 3; do docker run -d --name sm-agent-$i --hostname sm-agent-$i --network sm-cluster \ -e SM_AGENT_API_TOKEN \ grafana/synthetic-monitoring-agent:latest \ --api-server-address="$API_SERVER" \ --verbose=true \ --cluster-enabled \ --cluster-join-addresses=sm-agent-1,sm-agent-2,sm-agent-3 \ --cluster-wait-for-size=3 \ --cluster-wait-timeout=5m done ``` In this example, each agent waits up to 5 minutes for all three agents to join before it runs checks. To run agents on different hosts, publish the cluster port with `-p 7946:7946` and set `--cluster-advertise-address` to an address of the host that the other agents can reach. ## Configure clustering on Linux packages If you installed the agent from the [Debian](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#install-the-agent-on-ubuntudebian) or [RPM](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#install-the-agent-on-distributions-using-yum) packages, add the cluster flags to the agent’s systemd service on each host. 1. Open an override file for the service: Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo systemctl edit synthetic-monitoring-agent ``` 2. Add the following content, then save the file: ini ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```ini [Service] ExecStart= ExecStart=/usr/bin/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-insecure=${API_INSECURE} --listen-address=${LISTEN_ADDRESS} --verbose=${VERBOSE} --debug=${DEBUG} --features=${FEATURES} --cluster-enabled --cluster-join-addresses=probe-host-1,probe-host-2,probe-host-3 --cluster-wait-for-size=2 ``` The empty `ExecStart=` line clears the command from the packaged service, so the second line must repeat the packaged flags before adding the cluster flags. Replace `probe-host-1,probe-host-2,probe-host-3` with the hostnames or IP addresses of the agents in the cluster. 3. Restart the agent: Bash ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy ```bash sudo systemctl restart synthetic-monitoring-agent ``` Repeat these steps on every host in the cluster. By default, the agent advertises the first address it finds on the `eth0` or `en0` interface. Many Linux hosts use other interface names, such as `ens3` or `enp0s3`, and the agent doesn’t start if it can’t find an address to advertise. On these hosts, also set `--cluster-advertise-interfaces` to the name of your interface, or `--cluster-advertise-address` to the address of the host. ## Cluster flags reference Expand table | Flag | Default | Description | |---------------------------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-cluster-enabled` | `false` | Starts the agent in clustering mode. | | `-cluster-node-name` | Hostname | Name of this agent in the cluster. Must be unique within the cluster. | | `-cluster-name` | `""` | Name of the cluster. Agents with a different cluster name can’t join this cluster, which prevents separate clusters from merging by accident. | | `-cluster-join-addresses` | `""` | Comma-separated list of addresses to join. Refer to [Join address formats](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/clustering/#join-address-formats). | | `-cluster-listen-port` | `7946` | TCP port the agent listens on for cluster traffic. | | `-cluster-advertise-address` | `""` | Address, as `host[:port]`, that other agents use to reach this agent. The port defaults to `-cluster-listen-port`. If you don’t set it, the agent picks an address from `-cluster-advertise-interfaces`. | | `-cluster-advertise-interfaces` | `eth0,en0` | Comma-separated list of network interfaces to pick the advertise address from when `-cluster-advertise-address` isn’t set. | | `-cluster-wait-for-size` | `0` | Number of agents, including this one, that the cluster must reach before this agent runs checks. `0` or `1` turns off waiting. | | `-cluster-wait-timeout` | `60s` | Maximum time to wait for `-cluster-wait-for-size` before running checks with the agents that are available. `0` waits forever. | | `-cluster-rejoin-interval` | `60s` | How often the agent resolves `-cluster-join-addresses` again and rejoins the peers it finds. Rejoining reconnects agents that ended up in separate clusters, for example after a network partition. `0` turns off rejoining, so peers are only discovered at startup. | ### Join address formats `-cluster-join-addresses` accepts IP addresses and DNS names, each with an optional port. If you leave out the port, the agent uses `-cluster-listen-port`, so all agents must listen on the same port. In most cases, you don’t need a prefix. A DNS name resolves to all of its A/AAAA records, and falls back to SRV records if it has none. For example, a Kubernetes headless Service name resolves to the IP addresses of all its Pods. To choose how a DNS name is resolved, for example when your DNS publishes the agents only as SRV records, add one of the following prefixes: Expand table | Prefix | Lookup | |--------------|---------------------------------------------------------| | `dns+` | A/AAAA records only, without falling back to SRV. | | `dnssrv+` | SRV records, then A/AAAA records for each SRV target. | | `dnssrvnoa+` | SRV records only. The SRV targets are used as they are. | The agent ignores ports in SRV records. It always uses the port from the address, or `-cluster-listen-port`. If an address doesn’t resolve, the agent logs a warning and skips it. The agent only fails to join when none of the addresses resolve. ## Cluster states and lifecycle Each agent in the cluster is in one of the following states: - **Viewer:** The agent has a read-only view of the cluster and doesn’t own any checks. - **Participant:** The agent owns and runs its share of the probe’s checks. - **Terminating:** The agent is shutting down and no longer owns any checks. An agent joins the cluster as a viewer and becomes a participant once it has started. It moves to terminating when it shuts down. - **Startup:** If none of the join addresses resolve, or the agent can’t join any peer, the agent starts as a single-node cluster and tries to rejoin every `-cluster-rejoin-interval`. While it waits for `-cluster-wait-for-size`, the agent receives checks but doesn’t run them. - **Graceful shutdown:** When the agent receives `SIGTERM`, it marks itself as terminating and stays in the cluster for a few seconds, so the other agents can take over its checks. - **Agent failure:** If an agent stops without shutting down gracefully, the other agents detect the failure after a few seconds and take over its checks. Until then, that agent’s checks don’t run. > Warning > > Clustering is eventually consistent. While agents join or leave, or during a network partition, a check might briefly run on two agents or not run at all. During a network partition, each side of the partition runs the full set of checks until the partition heals. > Warning > > Traffic between agents isn’t encrypted or authenticated. Run clustered agents on a trusted network, and only allow cluster traffic between the agents. Ad-hoc checks, such as the ones you run with the **Test** button in the check editor, don’t use cluster ownership. Synthetic Monitoring sends each of these checks to a single agent in the cluster, and that agent runs it once. To upgrade the agents in a cluster, refer to [Keep the agent up to date](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/set-up/#keep-the-agent-up-to-date). ## Monitor the cluster When clustering is enabled, each agent exposes the following metrics on its metrics endpoint. Refer to [Monitor private probes](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/monitor-private-probes/). Like the rest of clustering, these metrics might change while the feature is experimental. ### Agent metrics Expand table | Metric | Type | Description | |------------------------------------------------|---------|------------------------------------------------------------------| | `sm_agent_cluster_size` | Gauge | Number of agents in the cluster, including this one. | | `sm_agent_cluster_ring_ready` | Gauge | `1` once the agent is ready to run checks, `0` otherwise. | | `sm_agent_cluster_peer_resolve_failures_total` | Counter | Number of times resolving `-cluster-join-addresses` failed. | | `sm_agent_cluster_join_failures_total` | Counter | Number of times joining the discovered peers failed. | | `sm_agent_updater_known_checks` | Gauge | Number of checks the agent received for the probe, owned or not. | | `sm_agent_updater_owned_checks` | Gauge | Number of checks this agent owns and runs. | ### Gossip metrics These metrics come from the gossip layer that the agents use to form the cluster. Each of these metrics has a `cluster_name` label with the value of `-cluster-name`. Expand table | Metric | Type | Description | |---------------------------------------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------| | `cluster_node_peers` | Gauge | Number of healthy peers seen by this agent, by `state`: `viewer`, `participant`, or `terminating`. | | `cluster_node_gossip_health_score` | Gauge | Health of this agent’s gossip communication. `0` is healthy. Higher values mean the agent has trouble communicating with its peers. | | `cluster_node_gossip_received_events_total` | Counter | Number of gossip messages handled by this agent, by `event`. The `node_conflict` event means that another agent tried to join with the same node name. | In a healthy cluster: - `sm_agent_cluster_size` is the same on every agent and equals the number of agents you run. - `sm_agent_cluster_ring_ready` is `1` on every agent. - `sm_agent_updater_known_checks` is the same on every agent. - The sum of `sm_agent_updater_owned_checks` across agents equals `sm_agent_updater_known_checks`. - `cluster_node_gossip_health_score` is `0` on every agent. - `cluster_node_peers{state="terminating"}` doesn’t stay above `0`. - `cluster_node_gossip_received_events_total{event="node_conflict"}` doesn’t increase. For help with a cluster that doesn’t form, refer to [Troubleshoot private probes](https://grafana.com/docs/grafana-cloud/observe-and-act/testing/synthetic-monitoring/set-up/set-up-private-probes/troubleshoot/).