Configure clustering on Grafana Labs

Configure Alloy for clustering

Fri, 25 Sep 2026 13:16:18 +0000

Configure Alloy for clustering

You can enable clustering in Alloy with Helm chart settings or command-line flags.

Note
Cluster mode in Alloy allows instances to discover each other and form a cluster. To distribute workload, you must also [enable clustering in individual components][distribute-workload].

Configure clustering with a Helm chart

You can enable clustering when Alloy is installed on Kubernetes with a Helm chart.

Before you begin

For multi-replica clustering on Kubernetes, set controller.type to statefulset. A StatefulSet gives each Pod a stable network identity for peer discovery. It also lets you choose how many Alloy Pods to run instead of running one Pod on every node. Use a DaemonSet only when your configuration must collect node-local data.

Steps

To configure clustering:

Amend your existing values.yaml file to add clustering.enabled=true inside the alloy block:
YAML
```
alloy:
  clustering:
    enabled: true
```
Upgrade your installation to use the new values.yaml file:
Bash
```
helm upgrade --namespace <NAMESPACE> <RELEASE_NAME> grafana/alloy -f values.yaml
```
Replace the following:
- <NAMESPACE>: The Kubernetes namespace where you installed the Helm chart.
- <RELEASE_NAME>: The name of the installation you chose when you installed the Helm chart.
Use the Alloy UI to verify the cluster status:
1. Click Clustering in the navigation bar.
2. Ensure that all expected nodes appear in the resulting table.

Configure clustering with command-line flags

If you run Alloy outside of Kubernetes or without the Helm chart, use command-line flags to enable clustering.

Required flags

Pass --cluster.enabled and one of the following peer-discovery flags to the alloy run command:

Flag	Description
`--cluster.enabled`	Enables clustering mode.
`--cluster.join-addresses`	Comma-separated list of addresses of cluster nodes to join.
`--cluster.discover-peers`	Key-value tuples used to discover peers dynamically.

Example

alloy run config.alloy \
  --cluster.enabled \
  --cluster.join-addresses=alloy-1:7946,alloy-2:7946

The first node that bootstraps a new cluster can omit peer discovery flags or connect to itself.

Cluster peers communicate over HTTP/2 on the built-in HTTP server. Each node must accept connections on --server.http.listen-addr and on the address defined or inferred by --cluster.advertise-address. If --cluster.advertise-address isn’t set, Alloy tries to infer an address from --cluster.advertise-interfaces. The default interfaces are eth0 and en0. On Windows, set --cluster.advertise-interfaces to a valid network interface or set --cluster.advertise-address explicitly. Alloy fails to start if it can’t determine the advertised address.

Optional flags

You can customize clustering behavior with additional flags:

Flag	Description	Default
`--cluster.advertise-address`	Address to advertise to other cluster nodes.	`""`
`--cluster.advertise-interfaces`	Network interfaces to use for advertisement.	`eth0,en0`
`--cluster.enable-tls`	Enables TLS for cluster communication.	`false`
`--cluster.max-join-peers`	Number of peers to join from the discovered set.	`5`
`--cluster.name`	Cluster name that nodes must share to join the same cluster.	`""`
`--cluster.node-name`	The name to use for this node.	`""`
`--cluster.rejoin-interval`	Interval to rejoin the cluster.	`60s`
`--cluster.tls-ca-path`	Path to the CA certificate file.	`""`
`--cluster.tls-cert-path`	Path to the certificate file.	`""`
`--cluster.tls-key-path`	Path to the key file.	`""`
`--cluster.tls-server-name`	Server name to use for TLS communication.	`""`
`--cluster.wait-for-size`	Minimum cluster size before traffic processing.	`0`
`--cluster.wait-timeout`	Timeout for cluster size wait.	`0`

To enable TLS for peer-to-peer communication, set --cluster.enable-tls and configure the related --cluster.tls-* flags for the CA certificate, certificate file, key file, and server name.

The --cluster.rejoin-interval flag controls how often each node rediscovers peers from --cluster.join-addresses and --cluster.discover-peers and tries to rejoin them. This can help recover from split-brain conditions and support dynamic environments. To discover peers only at startup, set --cluster.rejoin-interval="0s". After startup, nodes use gossip messages to converge on the cluster state.

The --cluster.max-join-peers flag limits how many discovered peers a node tries to connect to when joining or rejoining a cluster. This is useful for large clusters where connecting to many peers can be expensive. To disable the limit, set --cluster.max-join-peers=0. If --cluster.max-join-peers is higher than the number of discovered peers, Alloy connects to all discovered peers.

The --cluster.name flag prevents clusters from accidentally merging. Nodes only join peers that use the same cluster name. If a node tries to join a cluster with the wrong --cluster.name, it returns a failed to join memberlist error.

For --cluster.join-addresses DNS prefix syntax and --cluster.discover-peers tuple quoting rules, refer to the alloy run reference.

For production deployments, set --cluster.wait-for-size to your expected cluster size and --cluster.wait-timeout to a reasonable duration. This ensures all nodes join before processing begins, which reduces duplicate scrapes and out-of-order samples at remote write during startup.

If you leave --cluster.wait-for-size at the default 0, nodes can scrape before enough peers have joined, so two nodes may write the same series and trigger errors. Refer to Out of order errors for additional troubleshooting.

The following example configures a 3-node cluster where each node waits up to 5 minutes for all cluster members to join before it starts processing traffic:

alloy run config.alloy \
   --cluster.enabled \
   --cluster.join-addresses=alloy-1:7946,alloy-2:7946,alloy-3:7946 \
   --cluster.wait-for-size=3 \
   --cluster.wait-timeout=5m

Refer to the alloy run reference for complete details on all clustering flags.

Verify cluster status

After you enable clustering, verify that all nodes have joined the cluster:

Open the Alloy UI on any cluster node.
Click Clustering in the navigation bar.
Verify that all expected nodes appear in the cluster members table.

Next steps

After you enable clustering in Alloy, configure components to distribute workload.

Distribute workload across cluster nodes

Fri, 25 Sep 2026 13:16:18 +0000

Distribute workload across cluster nodes

After you enable clustering in Alloy, you can configure individual components to distribute their workload across the cluster.

Clustering with target auto-distribution allows a fleet of Alloy instances to dynamically distribute workload and provides high availability and horizontal scalability.

Before you begin

Configure clustering in your Alloy installation.
Ensure that all clustered Alloy instances have the same configuration file.

Enable clustering in components

To enable workload distribution, add a clustering block with enabled = true inside each component block that should participate:

component.type "example" {
  # ...component settings...

  clustering {
    enabled = true
  }
}

Note
Components don’t automatically participate in clustering. You must explicitly enable clustering in each component that should distribute workload. If cluster mode is enabled but a component doesn’t have clustering { enabled = true }, every node runs that component’s full workload, which causes duplicate scraping, log collection, or profiling.

Components that support clustering

The following components support the clustering block:

Prometheus metrics collection components:

Pyroscope profiling components:

pyroscope.scrape

Loki log collection components:

Database observability components:

database_observability.mysql

Example: Distribute Prometheus metrics scrape load

This example shows how to configure prometheus.scrape to distribute scrape targets across cluster nodes.

Add a clustering block to your prometheus.scrape component:

prometheus.scrape "default" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.remote_write.default.receiver]

  clustering {
    enabled = true
  }
}

Restart or reload Alloy for it to use the new configuration.
Validate that auto-distribution works:
1. Use the Alloy UI on each node, navigate to the details page for the prometheus.scrape component.
2. Compare the Debug Info sections between two different nodes to ensure that they don’t scrape the same sets of targets.

Example: Distribute log collection

This example shows how to configure loki.source.kubernetes to distribute log collection across cluster nodes.

loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.write.default.receiver]

  clustering {
    enabled = true
  }
}

Example: Distribute profiling targets

This example shows how to configure pyroscope.scrape to distribute profiling targets across cluster nodes.

pyroscope.scrape "default" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [pyroscope.write.default.receiver]

  clustering {
    enabled = true
  }
}

How workload distribution works

When you enable clustering in a component:

All cluster nodes use a consistent hashing algorithm to determine target ownership.
Each node only processes the subset of targets it’s responsible for.
When a node joins or leaves the cluster, targets are automatically redistributed.
Approximately 1/N of targets are redistributed when cluster membership changes. This minimizes disruption.

Note
Target labels must be consistent across all cluster nodes. If target discovery adds node-specific labels, such as constants.hostname, the hashing algorithm can’t consistently determine target ownership.

For more information about clustering concepts, refer to Clustering.