This is documentation for the next version of Grafana Mimir documentation. For the latest stable release, go to the latest version.

Documentation Grafana Mimir References Advanced architecture Components Compactor-scheduler

Grafana Mimir compactor-scheduler

The compactor-scheduler is an optional component that coordinates compaction work across compactors. It maintains a queue of compaction jobs for each tenant and distributes the jobs to compactors over gRPC.

When a compactor-scheduler is deployed, compactors run in scheduler mode: instead of using a hash ring to determine which tenants and blocks to compact, they request jobs from the compactor-scheduler and execute them. Any compactor can execute jobs for any tenant. For a description of standalone mode, which is the default, refer to compactor sharding.

Benefits of using the compactor-scheduler

Compared to standalone mode, scheduler mode:

• Distributes work dynamically through queuing rather than through a hash ring. If a job is available, any free compactor can lease it.
• Deduplicates compaction planning across compactor replicas: planned compaction jobs are shared through the compactor-scheduler. This reduces the number of object storage operations required for compaction planning.
• Separates compaction planning from execution, so compaction can progress even if planning degrades.
• Caches block metadata in a shared external cache. In standalone mode, each compactor caches block metadata on its own disk, so scaling events cause metadata to be refetched from object storage.
• Enables autoscaling compactors based on direct compaction job information, such as total bytes of pending work, rather than second-order signals like CPU usage.
• Retries failed jobs without re-planning the whole tenant.

How it works

The following flow describes how compaction work moves through a Grafana Mimir cluster running in scheduler mode:

1. The compactor-scheduler discovers the tenants by listing the object storage bucket, at every interval defined by -compactor-scheduler.tenant-discovery-interval.
2. For each tenant, the compactor-scheduler enqueues a planning job at every interval defined by -compactor-scheduler.planning-interval.
3. A compactor leases the planning job, computes the compaction plan for the tenant, and returns the resulting compaction jobs to the compactor-scheduler, which enqueues them. The compactor-scheduler never reads blocks or the bucket index itself.
4. Compactors lease the compaction jobs, execute them, and periodically report progress back to the compactor-scheduler.
5. When a compactor reports that a job completed, the compactor-scheduler removes the job from the queue.

Planning jobs and compaction jobs are kept in separate queues, called lanes. Within each lane, the compactor-scheduler ensures tenant fairness using a round-robin across all tenants that have pending jobs in that lane.

The -compactor.scheduler-client.lanes parameter configures the worker goroutines each compactor runs and the lanes each worker leases jobs from. The default value, compact+plan,plan, runs two workers: one that leases compaction jobs and falls back to planning jobs, and one dedicated to planning jobs, so that planning is not starved by long-running compactions. Each worker executes one job at a time, so the compaction parallelism of a compactor is determined by how many of its workers lease compaction jobs.

Job leases

Jobs are leased to compactors, not permanently assigned. If a compactor doesn’t report progress on a job for longer than -compactor-scheduler.lease-duration, the compactor-scheduler makes the job available for other compactors to lease. A job that has been leased more than -compactor-scheduler.repeated-failure-report-threshold times without completing is reported as a repeated failure, and once it has been leased -compactor-scheduler.max-leases times it is removed from the queue. Planning jobs are exempt from these limits and are always retried. A discarded job is still re-planned and re-enqueued on the next planning interval, unless it no longer exists (for example, if the tenant has been deleted or the blocks were marked for deletion).

Block metadata cache

Both planning jobs and compaction jobs read block metadata from object storage. Compactors can cache this metadata in a shared external cache, configured with the -compactor.scheduler-client.metadata-cache.* parameters, to reduce object storage operations.

State

The compactor-scheduler persists job information to local disk, in bbolt databases stored under -compactor-scheduler.bbolt.dir. The state is sharded across multiple database files, configured by -compactor-scheduler.bbolt.shard-count. Changing the shard count triggers an automatic migration of the state at the next startup.

After a restart, the compactor-scheduler recovers state from disk. When it starts with no recovered state, it delays planning for a few maintenance intervals, defined by -compactor-scheduler.maintenance-intervals-before-cold-start-planning, to avoid enqueuing jobs that duplicate work still in progress. After every restart, it also waits a few maintenance intervals, defined by -compactor-scheduler.maintenance-intervals-before-lease-expiration, before enforcing lease expiration to provide a grace period for compactors. If the persisted state becomes corrupted, it can be wiped and rebuilt: refer to the recovery steps in the runbooks.

Run exactly one compactor-scheduler replica. Two active compactor-schedulers would both schedule work for all tenants, resulting in duplicate compactions. Because it runs as a single replica, zone-aware replication doesn’t apply to the compactor-scheduler.

A short compactor-scheduler outage doesn’t stop in-progress compactions: compactors keep executing the jobs they hold, retry job requests with backoff, and resume progress reports at the next report interval, until the compactor-scheduler is available again.

Enable scheduler mode

To run compaction in scheduler mode:

1. Deploy the compactor-scheduler, using -target=compactor-scheduler and other configuration parameters as described in compactor-scheduler configuration.
2. Configure the compactors with:
   • -compactor.scheduler-client.enabled=true
   • -compactor.scheduler-client.scheduler-endpoint=<host:port>, pointing to the compactor-scheduler gRPC endpoint.

In scheduler mode, compactors still register in the compactor hash ring to coordinate blocks cleanup, which includes keeping the bucket index updated, deleting blocks, and enforcing retention.

Migrate from standalone mode

The mode is a per-compactor setting, applied at startup. There is no coordination between the two modes. Running both modes simultaneously could result in a tenant being planned and compacted by both. This doesn’t corrupt data, but it duplicates work for compactors and potentially increases store-gateway load. A mixed fleet is expected while the configuration change rolls out, but don’t run it as a steady state.

On a fresh deployment, the compactor-scheduler delays planning for a few maintenance intervals (-compactor-scheduler.maintenance-intervals-before-cold-start-planning), giving the compactor rollout time to complete before the first jobs are planned. The rollout itself stops any in-flight standalone compactions, so in practice little or no work is duplicated. This makes it safe to migrate in a single step, rolling out the compactor-scheduler and switching the compactors to scheduler mode together, as described above.

If limiting which tenants are compacted using -compactor.enabled-tenants or -compactor.disabled-tenants, the same values should be configured on the compactor-scheduler. Compactors should still be configured with the same values as they apply to blocks cleanup as well.

Compactor-scheduler configuration

Refer to the compactor_scheduler block in the Grafana Mimir configuration parameters for details of compactor-scheduler configuration, and to the scheduler_client block within the compactor block section for the compactor side of the configuration.