Documentation Grafana Alloy Configure Clustering Configure Alloy

Configure Alloy for clustering

You can enable clustering in Alloy with Helm chart settings or command-line flags.

Configure clustering with a Helm chart

You can enable clustering when Alloy is installed on Kubernetes with a Helm chart.

Before you begin

For multi-replica clustering on Kubernetes, set controller.type to statefulset. A StatefulSet gives each Pod a stable network identity for peer discovery. It also lets you choose how many Alloy Pods to run instead of running one Pod on every node. Use a DaemonSet only when your configuration must collect node-local data.

Steps

To configure clustering:

1. Amend your existing values.yaml file to add clustering.enabled=true inside the alloy block:

   YAML Copy

   alloy:
     clustering:
       enabled: true

2. Upgrade your installation to use the new values.yaml file:

   Bash Copy

   helm upgrade --namespace <NAMESPACE> <RELEASE_NAME> grafana/alloy -f values.yaml

   Replace the following:

   • <NAMESPACE>: The Kubernetes namespace where you installed the Helm chart.
   • <RELEASE_NAME>: The name of the installation you chose when you installed the Helm chart.

3. Use the Alloy UI to verify the cluster status:

   1. Click Clustering in the navigation bar.

   2. Ensure that all expected nodes appear in the resulting table.

Configure clustering with command-line flags

If you run Alloy outside of Kubernetes or without the Helm chart, use command-line flags to enable clustering.

Required flags

Pass --cluster.enabled and one of the following peer-discovery flags to the alloy run command:

Example

Bash Copy

alloy run config.alloy \
  --cluster.enabled \
  --cluster.join-addresses=alloy-1:7946,alloy-2:7946

The first node that bootstraps a new cluster can omit peer discovery flags or connect to itself.

Cluster peers communicate over HTTP/2 on the built-in HTTP server. Each node must accept connections on --server.http.listen-addr and on the address defined or inferred by --cluster.advertise-address. If --cluster.advertise-address isn’t set, Alloy tries to infer an address from --cluster.advertise-interfaces. The default interfaces are eth0 and en0. On Windows, set --cluster.advertise-interfaces to a valid network interface or set --cluster.advertise-address explicitly. Alloy fails to start if it can’t determine the advertised address.

Optional flags

You can customize clustering behavior with additional flags:

To enable TLS for peer-to-peer communication, set --cluster.enable-tls and configure the related --cluster.tls-* flags for the CA certificate, certificate file, key file, and server name.

The --cluster.rejoin-interval flag controls how often each node rediscovers peers from --cluster.join-addresses and --cluster.discover-peers and tries to rejoin them. This can help recover from split-brain conditions and support dynamic environments. To discover peers only at startup, set --cluster.rejoin-interval="0s". After startup, nodes use gossip messages to converge on the cluster state.

The --cluster.max-join-peers flag limits how many discovered peers a node tries to connect to when joining or rejoining a cluster. This is useful for large clusters where connecting to many peers can be expensive. To disable the limit, set --cluster.max-join-peers=0. If --cluster.max-join-peers is higher than the number of discovered peers, Alloy connects to all discovered peers.

The --cluster.name flag prevents clusters from accidentally merging. Nodes only join peers that use the same cluster name. If a node tries to join a cluster with the wrong --cluster.name, it returns a failed to join memberlist error.

For --cluster.join-addresses DNS prefix syntax and --cluster.discover-peers tuple quoting rules, refer to the alloy run reference.

For production deployments, set --cluster.wait-for-size to your expected cluster size and --cluster.wait-timeout to a reasonable duration. This ensures all nodes join before processing begins, which reduces duplicate scrapes and out-of-order samples at remote write during startup.

If you leave --cluster.wait-for-size at the default 0, nodes can scrape before enough peers have joined, so two nodes may write the same series and trigger errors. Refer to Out of order errors for additional troubleshooting.

The following example configures a 3-node cluster where each node waits up to 5 minutes for all cluster members to join before it starts processing traffic:

Bash Copy

alloy run config.alloy \
   --cluster.enabled \
   --cluster.join-addresses=alloy-1:7946,alloy-2:7946,alloy-3:7946 \
   --cluster.wait-for-size=3 \
   --cluster.wait-timeout=5m

Refer to the alloy run reference for complete details on all clustering flags.

Verify cluster status

After you enable clustering, verify that all nodes have joined the cluster:

1. Open the Alloy UI on any cluster node.
2. Click Clustering in the navigation bar.
3. Verify that all expected nodes appear in the cluster members table.

Next steps

After you enable clustering in Alloy, configure components to distribute workload.