Documentation Grafana Cloud Observe and act Test and validate Synthetic Monitoring Set up Private probes Clustering

Run private probes in a cluster

Clustering lets several agents share one private probe and split the probe’s checks between them. The agents form the cluster with a gossip protocol, and the cluster is eventually consistent. To learn how this affects your checks, refer to Cluster states and lifecycle. A non-clustered agent behaves the same as a single-node cluster.

You configure clustering with -cluster-* command-line flags. Before you set up a cluster, set up a private probe and install or deploy the agent. To decide whether you need a cluster, refer to Choose a deployment model.

How checks are distributed

Every agent in the cluster connects to Synthetic Monitoring with the same probe token and receives the full set of checks assigned to the probe. Each agent then runs only the checks that it owns.

The agents use consistent hashing to decide which agent owns each check. Each agent calculates ownership locally from the current cluster membership, so check ownership is never sent over the network.

When an agent joins or leaves the cluster, the agents recalculate ownership. Only about 1/N of the checks move, where N is the number of agents, which keeps disruption small. When an agent leaves or fails, the remaining agents pick up its checks.

Every agent reports its results under the same probe, so the probe looks the same in Synthetic Monitoring whether it runs on one agent or many.

Cluster requirements

Every agent in the cluster must:

• Use the same probe authentication token.
• Run the same version of the agent.
• Have the same feature flags enabled.
• Use the same image variant, for example the -browser image.
• Use the same browser pool configuration.
• Use the same -cluster-listen-port and the same -cluster-name.
• Accept TCP connections from the other agents on the cluster listen port, 7946 by default. Refer to Cluster traffic.
• Have a unique node name. The node name defaults to the hostname.

Check ownership doesn’t take an agent’s capabilities into account. If an agent that can’t run a scripted or browser check owns that check, the check doesn’t run on any agent.

On Kubernetes, deploy the agents as a Deployment with a headless Service. The headless Service lets agents discover each other.

Best practices

Plan for heavy checks

The cluster balances the number of checks across agents, not their cost. Check types use very different amounts of resources: a browser check uses far more CPU and memory than an HTTP check. If a probe runs a few heavy checks among many light ones, some agents can end up with more load than others. In this case, give each agent more CPU and memory, or move the heavy checks to a separate probe.

Browser checks use the most resources, so they’re the most likely to make some agents busier than others. If the probe runs more than a few browser checks, run them on a browser pool. The agents then do similar amounts of work for each check, so load stays more even across the cluster.

Size the cluster for peak load

Checks with long frequencies, such as hourly browser checks, run in bursts. Autoscalers react too slowly to these bursts, and scaling down moves checks to other agents, which then run those checks again right away. For probes that run heavy checks at long frequencies, run a fixed number of agents that can handle the peak load instead of autoscaling aggressively.

For sizing guidance, refer to Size clustered agents and browser pools.

Use -cluster-wait-for-size with care

When a new agent starts, it waits for the cluster to reach -cluster-wait-for-size agents before it runs any checks, so it doesn’t briefly run checks that other agents already own. If the cluster doesn’t reach that size within -cluster-wait-timeout, the agent runs checks anyway with the agents that are available.

Set -cluster-wait-for-size well below your usual number of agents, for example below your HorizontalPodAutoscaler or PodDisruptionBudget minimum. Keep -cluster-wait-timeout set. If it’s 0, an agent can wait forever and never run checks.

Only enable clustering when you need it

If a single agent can handle all the checks assigned to the probe, run a single agent.

Configure clustering on Kubernetes

These steps extend Deployment with Kubernetes. They reuse the synthetic-monitoring namespace and the sm-agent-1 secret from that section. Instead of deployment.yaml, they use the cluster files from the Kubernetes example in the agent GitHub repository.

1. Get the cluster deployment files.

   Bash Copy

   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/cluster-service.yaml
   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/cluster-deployment.yaml

   cluster-service.yaml defines a headless Service so agents can discover each other. cluster-deployment.yaml defines a Deployment that runs three agents.

   In these files:

   • --cluster-join-addresses points to the headless Service. The name resolves to the IP addresses of the agent Pods.
   • publishNotReadyAddresses: true includes Pods that aren’t Ready yet in the Service’s DNS records. If you add a readiness probe to the agents, this setting lets starting agents find peers that haven’t passed the probe yet, instead of each agent starting as a single-node cluster.
   • The node name defaults to the Pod hostname.
   • --cluster-wait-for-size=2 is below the number of replicas, so agents can still start when one Pod is unavailable.
   • terminationGracePeriodSeconds leaves time for an agent to hand its checks over to the other agents before it stops. Keep it above 5 seconds.

2. If you deployed a single agent with deployment.yaml, delete it first. The API rejects the clustered agents while the single agent is connected. The probe is offline until the cluster starts.

   Bash Copy

   kubectl delete deployment sm-agent-1 -n synthetic-monitoring

3. Deploy the headless Service and the agents.

   Bash Copy

   kubectl apply -f cluster-service.yaml
   kubectl apply -f cluster-deployment.yaml

4. Verify that the agents formed a cluster. Each agent logs the cluster members when it joins and whenever membership changes. List the agent Pods, then check the logs of one of them:

   Bash Copy

   kubectl get pods -n synthetic-monitoring
   kubectl logs -n synthetic-monitoring POD_NAME | grep "cluster peers changed"

   Replace POD_NAME with the name of one of the agent Pods, for example sm-agent-7c9d8b5f4-x2kqp.

   You can also check that the sm_agent_cluster_size metric on each agent equals the number of replicas. Refer to Monitor the cluster.

To change the number of agents, scale the Deployment. The agents redistribute checks automatically.

Configure clustering with Helm

If you deploy the agent with the Helm chart, enable clustering in your values file:

YAML Copy

replicas: 3
cluster:
  enabled: true
  # Keep it below replicas, so agents can still start when one Pod is unavailable.
  waitForSize: 2

For a new release, install the chart with this file. For an existing release, upgrade it:

Bash Copy

helm upgrade sm-agent oci://ghcr.io/grafana/helm-charts/synthetic-monitoring-agent \
  -n synthetic-monitoring -f values.yaml

The chart sets the join addresses to a headless Service, the advertise address to the Pod IP, and the cluster name to a name derived from the release. It also creates a PodDisruptionBudget, so that node drains evict one agent at a time. Optionally, the chart can also create a NetworkPolicy that only allows cluster traffic from the agents of the release. Refer to the chart README.

The chart can also add a HorizontalPodAutoscaler for the agents with autoscaling.enabled. Every scale event moves checks between agents, so size the minimum number of agents for peak load. If the probe runs more than a few browser checks, use a browser pool, so that agent load stays even. Refer to the chart README before you enable autoscaling.

Configure clustering with Docker

Pass -cluster-enabled and -cluster-join-addresses to each agent. The first agent that starts a new cluster can leave out -cluster-join-addresses.

Each agent advertises an address that the other agents use to reach it. By default, the agent uses the first address it finds on the eth0 or en0 interface. If the other agents can’t reach that address, for example when agents run on different hosts, set -cluster-advertise-address explicitly.

The following example runs three agents on the same host. It uses a user-defined Docker network so the containers can reach each other by name. Before you run it, export SM_AGENT_API_TOKEN and API_SERVER as described in Deploy the agent using Docker.

Bash Copy

docker network create sm-cluster

for i in 1 2 3; do
  docker run -d --name sm-agent-$i --hostname sm-agent-$i --network sm-cluster \
    -e SM_AGENT_API_TOKEN \
    grafana/synthetic-monitoring-agent:latest \
    --api-server-address="$API_SERVER" \
    --verbose=true \
    --cluster-enabled \
    --cluster-join-addresses=sm-agent-1,sm-agent-2,sm-agent-3 \
    --cluster-wait-for-size=3 \
    --cluster-wait-timeout=5m
done

In this example, each agent waits up to 5 minutes for all three agents to join before it runs checks.

To run agents on different hosts, publish the cluster port with -p 7946:7946 and set --cluster-advertise-address to an address of the host that the other agents can reach.

Configure clustering on Linux packages

If you installed the agent from the Debian or RPM packages, add the cluster flags to the agent’s systemd service on each host.

1. Open an override file for the service:

   Bash Copy

   sudo systemctl edit synthetic-monitoring-agent

2. Add the following content, then save the file:

   ini Copy

   [Service]
   ExecStart=
   ExecStart=/usr/bin/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-insecure=${API_INSECURE} --listen-address=${LISTEN_ADDRESS} --verbose=${VERBOSE} --debug=${DEBUG} --features=${FEATURES} --cluster-enabled --cluster-join-addresses=probe-host-1,probe-host-2,probe-host-3 --cluster-wait-for-size=2

   The empty ExecStart= line clears the command from the packaged service, so the second line must repeat the packaged flags before adding the cluster flags. Replace probe-host-1,probe-host-2,probe-host-3 with the hostnames or IP addresses of the agents in the cluster.

3. Restart the agent:

   Bash Copy

   sudo systemctl restart synthetic-monitoring-agent

Repeat these steps on every host in the cluster.

By default, the agent advertises the first address it finds on the eth0 or en0 interface. Many Linux hosts use other interface names, such as ens3 or enp0s3, and the agent doesn’t start if it can’t find an address to advertise. On these hosts, also set --cluster-advertise-interfaces to the name of your interface, or --cluster-advertise-address to the address of the host.

Cluster flags reference

Join address formats

-cluster-join-addresses accepts IP addresses and DNS names, each with an optional port. If you leave out the port, the agent uses -cluster-listen-port, so all agents must listen on the same port.

In most cases, you don’t need a prefix. A DNS name resolves to all of its A/AAAA records, and falls back to SRV records if it has none. For example, a Kubernetes headless Service name resolves to the IP addresses of all its Pods.

To choose how a DNS name is resolved, for example when your DNS publishes the agents only as SRV records, add one of the following prefixes:

The agent ignores ports in SRV records. It always uses the port from the address, or -cluster-listen-port.

If an address doesn’t resolve, the agent logs a warning and skips it. The agent only fails to join when none of the addresses resolve.

Cluster states and lifecycle

Each agent in the cluster is in one of the following states:

• Viewer: The agent has a read-only view of the cluster and doesn’t own any checks.
• Participant: The agent owns and runs its share of the probe’s checks.
• Terminating: The agent is shutting down and no longer owns any checks.

An agent joins the cluster as a viewer and becomes a participant once it has started. It moves to terminating when it shuts down.

• Startup: If none of the join addresses resolve, or the agent can’t join any peer, the agent starts as a single-node cluster and tries to rejoin every -cluster-rejoin-interval. While it waits for -cluster-wait-for-size, the agent receives checks but doesn’t run them.
• Graceful shutdown: When the agent receives SIGTERM, it marks itself as terminating and stays in the cluster for a few seconds, so the other agents can take over its checks.
• Agent failure: If an agent stops without shutting down gracefully, the other agents detect the failure after a few seconds and take over its checks. Until then, that agent’s checks don’t run.

Ad-hoc checks, such as the ones you run with the Test button in the check editor, don’t use cluster ownership. Synthetic Monitoring sends each of these checks to a single agent in the cluster, and that agent runs it once.

To upgrade the agents in a cluster, refer to Keep the agent up to date.

Monitor the cluster

When clustering is enabled, each agent exposes the following metrics on its metrics endpoint. Refer to Monitor private probes. Like the rest of clustering, these metrics might change while the feature is experimental.

Agent metrics

Gossip metrics

These metrics come from the gossip layer that the agents use to form the cluster. Each of these metrics has a cluster_name label with the value of -cluster-name.

In a healthy cluster:

• sm_agent_cluster_size is the same on every agent and equals the number of agents you run.
• sm_agent_cluster_ring_ready is 1 on every agent.
• sm_agent_updater_known_checks is the same on every agent.
• The sum of sm_agent_updater_owned_checks across agents equals sm_agent_updater_known_checks.
• cluster_node_gossip_health_score is 0 on every agent.
• cluster_node_peers{state="terminating"} doesn’t stay above 0.
• cluster_node_gossip_received_events_total{event="node_conflict"} doesn’t increase.

For help with a cluster that doesn’t form, refer to Troubleshoot private probes.