This is documentation for the next version of Tempo. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Manage Tune the consistent hash rings

Tune the consistent hash rings

Tempo uses the Consistent Hash Ring implementation from Cortex. By default, the ring is gossiped between all Tempo components. However, it can be configured to use Consul or Etcd, if desired.

There are four consistent hash rings: distributor, ingester, metrics-generator, and compactor. Each hash ring exists for a distinct reason.

Distributor

Participants: Distributors

Used by: Distributors

Unless you are running with limits, this ring does not impact Tempo operation.

This ring is only used when global rate limits are used. The distributors use it to count the other active distributors. Incoming traffic is assumed to be evenly spread across all distributors and (global_rate_limit / # of distributors) is used to rate limit locally.

Ingester

Participants: Ingesters

Used by: Distributors, Queriers

This ring is used by the distributors to load balance traffic into the ingesters. When spans are received the trace id is hashed and they are sent to the appropriate ingesters based on token ownership in the ring. Queriers also use this ring to find the ingesters for querying recent traces.

Metrics-generator

Participants: Metrics-generators

Used by: Distributors

This ring is used by distributors to load balance traffic to the metrics-generators. When spans are received, the trace ID is hashed, and the traces are sent to the appropriate metrics-generators based on token ownership in the ring.

Compactor

Participants: Compactors

Used by: Compactors

This ring is used by the compactors to shard compaction jobs. Jobs are hashed into the ring and the owning compactor is the only one allowed to compact a specific set of blocks to prevent race conditions on compaction.

Interacting with the rings

Web pages are available at the following endpoints. They show every ring member, their tokens and includes the ability to “Forget” a ring member. “Forgetting” is useful when a ring member leaves the ring without properly shutting down (and therefore leaves its tokens in the ring).

Distributor

Available on: Distributors

Path: /distributor/ring

Unhealthy distributors have little impact but should be forgotten to reduce cost of maintaining the ring .

Ingester

Available on: Distributors

Path: /ingester/ring

Unhealthy ingesters will cause writes to fail. If the ingester is really gone, forget it immediately.

Metrics-generators

Available on: Distributors

Path: /metrics-generator/ring

Unhealthy metrics-generators will cause writes to fail. If the metrics-generator is really gone, forget it immediately.

Compactor

Available on: Compactors

Path: /compactor/ring

Unhealthy compactors will allow the blocklist to grow significantly. If the compactor is really gone, forget it immediately.

Configuring the rings

Ring/Lifecycler configuration control how a component interacts with the ring. Refer to the configuration topic for more details.