This is documentation for the next version of Mimir distributed. For the latest stable release, go to the latest version.
Configure a unique Grafana Mimir’s Memberlist cluster label in the mimir-distributed Helm chart installation
This document shows the steps to configure cluster label verification in a Grafana Mimir installed by Helm. Multiple Memberlist gossip ring clusters are at risk of merging into one without enabling cluster label verification. For example, if a Mimir, Tempo or Loki are running in the same Kubernetes cluster, they might communicate with each other without this configuration update. Once cluster label verification is enabled, before Mimir components communicate with other components, they will verify whether the other components have the same cluster label. The process to update the configuration will take three rollouts of the whole cluster.
Before you begin
- You have a Grafana Mimir installed by mimir-distributed helm chart with its Memberlist cluster label still set to default value.
- You have
kubectl
andhelm
command line configured to connect to the Kubernetes cluster where your Grafana Mimir is running.
Configuration update steps
There are three steps of the configuration update:
- Disable Memberlist cluster label verification
- Set cluster label on all Mimir components
- Enable Memberlist cluster label verification again
1. Disable Memberlist cluster label verification
Cluster label verification flag is enabled by default with cluster label set to an empty string. Using the default value of cluster label can make different systems that use Memberlist communicate with each other if they also have not updated the default cluster label. Setting a new cluster label directly to a non-empty string value without first disabling cluster label verification will cause Memberlist to form partition in the Grafana Mimir cluster. The partition makes some Mimir components have different cluster label values which can prevent the component from communicating. To disable cluster label verification flag, set the following structured config in mimir-distributed values.yaml configuration.
mimir:
structuredConfig:
memberlist:
cluster_label_verification_disabled: true
Rollout the installation to apply the configuration changes by running helm upgrade <my-mimir-release> mimir-distributed -f values.yaml
.
Replace <my-mimir-release>
with the actual Mimir release name. Wait until all Pods are ready before going to the next step.
2. Set cluster label on all Mimir components
Set cluster label on all Mimir components by setting the following configuration.
The configuration will set cluster_label
to the Helm release name and the namespace where the helm release is installed.
Updating a new cluster label after disabling cluster label verification will prevent Memberlist from forming a partition.
mimir:
structuredConfig:
memberlist:
cluster_label_verification_disabled: true
cluster_label: "{{.Release.Name}}-{{.Release.Namespace}}"
Apply the configuration changes again by running helm upgrade <my-mimir-release> mimir-distributed -f values.yaml
.
Replace <my-mimir-release>
with the actual Mimir release name. Wait until all Pods are ready before going to the next step.
3. Enable Memberlist cluster label verification
Remove mimir.structuredConfig.memberlist.cluster_label_verification_disabled
from the values.yaml file to re-enable Memberlist cluster label verification.
mimir:
structuredConfig:
memberlist:
cluster_label: "{{.Release.Name}}-{{.Release.Namespace}}"
Apply the configuration changes by running helm upgrade <my-mimir-release> mimir-distributed -f values.yaml
.
Replace <my-mimir-release>
with the actual Mimir release name. Wait until all Pods are ready before verifying that the configuration is applied correctly.
Verifying the configuration changes
Once the rollout is completed, verify the change by looking at the /memberlist
endpoint in some of Grafana Mimir pods.
Run the following port-forward command on several different Grafana Mimir components.
kubectl port-forward pod/<mimir-pod-1> --kube-context=<my-k8s-context> --namespace=<my-mimir-namespace> 8080:8080
kubectl port-forward pod/<mimir-pod-2> --kube-context=<my-k8s-context> --namespace=<my-mimir-namespace> 8081:8080
Replace <mimir-pod-1>
and <mimir-pod-2>
with several actual pods from different Mimir components.
Ensure the host port 8080 and 8081 are available, otherwise use different available ports.
Open the port-forwarded URL in browser to see the Memberlist status http://localhost:8080/memberlist, http://localhost:8081/memberlist and also few others Grafana Mimir components. The Memberlist page from different pods must show same view of all of their members.