The polling cycle is controlled by a number of configuration options detailed here.
storage: trace: # How often to repoll the backend for new blocks. Default is 5m [blocklist_poll: <duration>] # Number of blocks to process in parallel during polling. Default is 50. [blocklist_poll_concurrency: <int>] # By default components will pull the blocklist from the tenant index. If that fails the component can # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true. [blocklist_poll_fallback: <bool>] # Maximum number of compactors that should build the tenant index. All other components will download # the index. Default 2. [blocklist_poll_tenant_index_builders: <int>] # The oldest allowable tenant index. If an index is pulled that is older than this duration, # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here. # If fallback is true and a tenant index exceeds this duration, it will fall back to listing # the bucket contents. # Default 0 (disabled). [blocklist_poll_stale_tenant_index: <duration>]
Due to the mechanics of the tenant index the blocklist will be stale by
at most 2 times the configured
blocklist_poll duration. There are two configuration options that need to be balanced
blockist_poll to handle this:
complete_block_timeout is used to hold a block in the ingester for a given period of time after
it has been flushed. This allows the ingester to return traces to the queriers while they are still unaware
of the newly flushed blocks.
ingester: # How long to hold a complete block in the ingester after it has been flushed to the backend. Default is 15m [complete_block_timeout: <duration>]
compacted_block_retention is used to keep a block in the backend for a given period of time
after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access
these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the
complete_block_timeout this should be at a minimum 2x the configurated
compactor: compaction: # How long to leave a block in the backend after it has been compacted successfully. Default is 1h [compacted_block_retention: <duration>]
Additionally, it is important that the querier
blocklist_poll duration is greater than or equal to the compactor
blocklist_poll duration. Otherwise a querier may not correctly check all assigned blocks and incorrectly return 404.
It is recommended to simply set both components to use the same poll duration.
Related Tempo resources
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.