This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Manage

Manage advanced systems

Tune the consistent hash rings

Open source

Tune the consistent hash rings

Tempo uses the Consistent Hash Ring implementation from Cortex. By default, the ring is gossiped between all Tempo components. However, it can be configured to use Consul or Etcd, if desired.

Tempo uses several consistent hash rings. This topic describes the distributor, live-store (partition ring), and live-store rings. Each hash ring exists for a distinct reason.

Distributor

Participants: Distributors

Used by: Distributors

Unless you are running with limits, this ring does not impact Tempo operation.

This ring is only used when global rate limits are used. The distributors use it to count the other active distributors. Incoming traffic is assumed to be evenly spread across all distributors and (global_rate_limit / # of distributors) is used to rate limit locally.

Live-store (partition ring)

Participants: Live-stores

Used by: Distributors, Queriers, Block-builders

The partition ring tracks which Tempo partitions are active and which live-stores own them. Distributors use this ring to determine which partitions to write to when sending data to Kafka. Queriers use it to find the live-stores for querying recent traces. Block-builders use it to determine which partitions to consume from.

Interacting with the rings

Web pages are available at the following endpoints. They show every ring member, their tokens and includes the ability to “Forget” a ring member. “Forgetting” is useful when a ring member leaves the ring without properly shutting down, and therefore leaves its tokens in the ring.