The polling cycle is controlled by a number of configuration options detailed here.
storage: trace: # How often to repoll the backend for new blocks. Default is 5m [blocklist_poll: <duration>] # Number of blocks to process in parallel during polling. Default is 50. [blocklist_poll_concurrency: <int>] # By default components will pull the blocklist from the tenant index. If that fails the component can # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true. [blocklist_poll_fallback: <bool>] # Maximum number of compactors that should build the tenant index. All other components will download # the index. Default 2. [blocklist_poll_tenant_index_builders: <int>] # The oldest allowable tenant index. If an index is pulled that is older than this duration, # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here. # If fallback is true and a tenant index exceeds this duration, it will fall back to listing # the bucket contents. # Default 0 (disabled). [blocklist_poll_stale_tenant_index: <duration>]
Due to the mechanics of the tenant index, the blocklist will be stale by
at most 2 times the configured
blocklist_poll duration. There are two configuration options that need to be balanced
blockist_poll to handle this:
complete_block_timeout is used to hold a block in the ingester for a given period of time after
it has been flushed. This allows the ingester to return traces to the queriers while they are still unaware
of the newly flushed blocks.
ingester: # How long to hold a complete block in the ingester after it has been flushed to the backend. Default is 15m [complete_block_timeout: <duration>]
compacted_block_retention is used to keep a block in the backend for a given period of time
after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access
these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the
complete_block_timeout, this should be at a minimum 2x the configured
compactor: compaction: # How long to leave a block in the backend after it has been compacted successfully. Default is 1h [compacted_block_retention: <duration>]
Additionally, the querier
blocklist_poll duration needs to be greater than or equal to the compactor
blocklist_poll duration. Otherwise, a querier may not correctly check all assigned blocks and incorrectly return 404.
It is recommended to simply set both components to use the same poll duration.