Menu
Documentation Grafana Mimir Configure Spread-minimizing tokens
Open source

Migrate ingesters to spread-minimizing tokens

Using this guide, you can configure Mimir’s ingesters to use the spread-minimizing token generation strategy.

Before you begin

The ingester time series replication should be configured with enabled zone-awareness.

Note

Spread-mimizing tokens are recommended if shuffle sharding is disabled in your ingesters, or, if shuffle-sharding is enabled, but most of the tenants of your system use all available ingesters.

For simplicity, let’s assume that there are three configured availability zones named zone-a, zone-b, and zone-c. Migration to the spread-minimizing token generation strategy is a complex process performed zone by zone to prevent any data loss.

Step 1: Disable write requests to ingesters from zone-a

To disable write requests, configure the following flag on the distributor and the ruler:

-ingester.ring.excluded-zones=zone-a

Before proceeding to the next step, use the following query to ensure that there are no write requests to the ingesters from zone-a:

sum by(route) (
  rate(
    cortex_request_duration_seconds_count{
      namespace="<your-namespace>",
      container="ingester",
      pod=~"ingester-zone-a-.*",
      route="/cortex.Ingester/Push"}[5m]
  )
)

You should see something like this:

No More Write Requests

Step 2: Shut down ingesters from zone-a

Next, ensure that all the in-memory series of all ingesters from zone-a have been flushed to long-term storage, as well as that the ingesters from zone-a have been forgotten from the ring. To do this, invoke the /ingester/shutdown endpoint on all the ingesters from zone-a.

Before proceeding to the next step, ensure that all the calls completed successfully completed.

Step 3: Enable spread-minimizing tokens for ingesters in zone-a

Configure the following flags on the ingesters from zone-a:

-ingester.ring.tokens-file-path=
-ingester.ring.token-generation-strategy=spread-minimizing
-ingester.ring.spread-minimizing-zones=zone-a,zone-b,zone-c

Note

The example uses zone-a,zone-b,zone-c to denote a comma-separated list of configured availability zones.

Before proceeding to the next step, ensure that all the ingester pods related to zone-a are up and running with the new configuration.

Optional step: In-order registration of ingesters

Mimir can force the ring to perform an in-order registration of ingesters. When this feature is enabled, an ingester can register its tokens within the ring only after all previous ingesters (with ID lower than its own ID) have already been registered. This feature minimizes a probability that a write request that should be handled by an ingester actually arrives to the ring before the ingester is registered within the ring. In this case, the request gets handled by another ingester. This situation could introduce some deviation from an optimal load distribution.

To configure this capability:

-ingester.ring.spread-minimizing-join-ring-in-order=true

Step 4: Re-enable write requests to ingesters from zone-a

To re-enable write requests, revert link Step 1: Disable write requests to ingesters from zone-a.

At this point, you can check the number of in-memory time series of ingesters from zone-a using the following query:

sum by(pod) (
  cortex_ingester_memory_series{
    namespace="<your-namespace>",
    pod=~"ingester-zone-a-.*"}
)

If everything went smoothly, you should see something like this:

Successful migration of ingesters from a zone

Step 5: Migrate ingesters from zone-b

Repeat steps 1 to 4, replacing all the occurrences of zone-a with zone-b.

Step 6: migrate ingesters from zone-c

Repeat steps 1 to 4, replacing all the occurrences of zone-a with zone-c.