Grafana Mimir memberlist and gossip protocol
Memberlist is a Go library that manages cluster membership, node failure detection, and message passing using a gossip-based protocol. Memberlist is eventually consistent and network partitions are partially tolerated by attempting to communicate to potentially dead nodes through multiple routes.
When using a memberlist-based KV store, each instance maintains a copy of the hash rings. Each Mimir instance updates a hash ring locally and uses memberlist to propagate the changes to other instances. Updates generated locally and updates received from other instances are merged together to form the current state of the ring on the instance.
To configure memberlist, refer to configuring hash rings.
How memberlist propagates hash ring changes
When using a memberlist-based KV store, every Grafana Mimir instance propagates the hash ring data structures to other instances using the following techniques:
- Propagating only the differences introduced in recent changes.
- Propagating the full hash ring data structure.
-memberlist.gossip-interval an instance randomly selects a subset of all Grafana Mimir cluster instances configured by
-memberlist.gossip-nodes and sends the latest changes to the selected instances.
This operation is performed frequently and it’s the primary technique used to propagate changes.
In addition, every
-memberlist.pullpush-interval an instance randomly selects another instance in the Grafana Mimir cluster and transfers the full content of the KV store, including all hash rings (unless
-memberlist.pullpush-interval is zero, which disables this behavior).
After this operation is complete, the two instances have the same content as the KV store.
This operation is computationally more expensive, and as a result, it’s performed less frequently. The operation ensures that the hash rings periodically reconcile to a common state.
Related Mimir resources
How to control metrics growth in Prometheus and Kubernetes with Grafana Cloud
This webinar will introduce a metrics cost management framework to optimize metrics growth while keeping rising costs at bay with Grafana Cloud.
Intro to Grafana Mimir: The open source time series database that scales to 1 billion metrics & beyond
Grafana Mimir webinar—learn about our open source solution for extending Prometheus at organizations needing massive scale, rapid query performance.
For billion-series scale or home IoT projects, get started in minutes with Grafana Mimir
Learn how easy it is to get started with Mimir, no matter how many or few time series you need to store.