Polling
The polling cycle is controlled by a number of configuration options detailed here.
storage:
trace:
# How often to repoll the backend for new blocks. Default is 5m
[blocklist_poll: <duration>]
# Number of blocks to process in parallel during polling. Default is 50.
[blocklist_poll_concurrency: <int>]
# By default components will pull the blocklist from the tenant index. If that fails the component can
# fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true.
[blocklist_poll_fallback: <bool>]
# Maximum number of compactors that should build the tenant index. All other components will download
# the index. Default 2.
[blocklist_poll_tenant_index_builders: <int>]
# The oldest allowable tenant index. If an index is pulled that is older than this duration,
# the polling will consider this an error. Note that `blocklist_poll_fallback` applies here.
# If fallback is true and a tenant index exceeds this duration, it will fall back to listing
# the bucket contents.
# Default 0 (disabled).
[blocklist_poll_stale_tenant_index: <duration>]
Due to the mechanics of the tenant index, the blocklist will be stale by
at most 2 times the configured blocklist_poll duration. There are two configuration options that need to be balanced
against the blockist_poll to handle this:
The ingester complete_block_timeout is used to hold a block in the ingester for a given period of time after
it has been flushed. This allows the ingester to return traces to the queriers while they are still unaware
of the newly flushed blocks.
ingester:
# How long to hold a complete block in the ingester after it has been flushed to the backend. Default is 15m
[complete_block_timeout: <duration>]
The compactor compacted_block_retention is used to keep a block in the backend for a given period of time
after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access
these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the
complete_block_timeout, this should be at a minimum 2x the configured blocklist_poll duration.
compactor:
compaction:
# How long to leave a block in the backend after it has been compacted successfully. Default is 1h
[compacted_block_retention: <duration>]
Additionally, the querier blocklist_poll duration needs to be greater than or equal to the compactor
blocklist_poll duration. Otherwise, a querier may not correctly check all assigned blocks and incorrectly return 404.
It is recommended to simply set both components to use the same poll duration.
