Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Configuration
You can use Grafana Cloud to avoid installing, maintaining, and scaling your own instance of Grafana Tempo. Create a free account to get started, which includes free forever access to 10k metrics, 50GB logs, 50GB traces, 500VUh k6 testing & more.
This document explains the configuration options for Tempo as well as the details of what they impact. It includes:
Use environment variables in the configuration
You can use environment variable references in the configuration file to set values that need to be configurable during deployment using --config.expand-env
option.
To do this, use:
${VAR}
Where VAR is the name of the environment variable.
Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the YAML file is parsed. References to undefined variables are replaced by empty strings unless you specify a default value or custom error text.
To specify a default value, use:
${VAR:-default_value}
where default_value is the value to use if the environment variable is undefined.
You can find more about other supported syntax here
Server
Tempo uses the Weaveworks/common server. For more information on configuration options, see here.
Distributor
For more information on configuration options, see here.
Distributors receive spans and forward them to the appropriate ingesters.
The following configuration enables all available receivers with their default configuration. For a production deployment, enable only the receivers you need. Additional documentation and more advanced configuration options are available in the receiver README.
Ingester
For more information on configuration options, see here.
The ingester is responsible for batching up traces and pushing them to TempoDB.
# Ingester configuration block
ingester:
# Lifecycler is responsible for managing the lifecycle of entries in the ring.
# For a complete list of config options check the lifecycler section under the ingester config at the following link -
# https://cortexmetrics.io/docs/configuration/configuration-file/#ingester_config
lifecycler:
ring:
# number of replicas of each span to make while pushing to the backend
replication_factor: 3
# amount of time a trace must be idle before flushing it to the wal.
# (default: 10s)
[trace_idle_period: <duration>]
# how often to sweep all tenants and move traces from live -> wal -> completed blocks.
# (default: 10s)
[flush_check_period: <duration>]
# maximum size of a block before cutting it
# (default: 1073741824 = 1GB)
[max_block_bytes: <int>]
# maximum length of time before cutting a block
# (default: 1h)
[max_block_duration: <duration>]
# duration to keep blocks in the ingester after they have been flushed
# (default: 15m)
[ complete_block_timeout: <duration>]
Query-frontend
For more information on configuration options, see here.
The Query Frontend is responsible for sharding incoming requests for faster processing in parallel (by the queriers).
# Query Frontend configuration block
query_frontend:
# number of times to retry a request sent to a querier
# (default: 2)
[max_retries: <int>]
# number of shards to split the query into
# (default: 20)
[query_shards: <int>]
# number of block queries that are tolerated to error before considering the entire query as failed
# numbers greater than 0 make possible for a read to return partial results
# partial results are indicated with HTTP status code 206
# (default: 0)
[tolerate_failed_blocks: <int>]
Querier
For more information on configuration options, see here.
The Querier is responsible for querying the backends/cache for the traceID.
# querier config block
querier:
# Timeout for trace lookup requests
[query_timeout: <duration> | default = 10s]
# Timeout for search requests
[search_query_timeout: <duration> | default = 30s]
# Limit used for search requests if none is set by the caller
[search_default_result_limit: <int> | default = 20]
# The maximum allowed value of the limit parameter on search requests. If the search request limit parameter
# exceeds the value configured here it will be set to the value configured here.
# The default value of 0 disables this limit.
[search_max_result_limit: <int> | default = 0]
# config of the worker that connects to the query frontend
frontend_worker:
# the address of the query frontend to connect to, and process queries
# Example: "frontend_address: query-frontend-discovery.default.svc.cluster.local:9095"
[frontend_address: <string>]
It also queries compacted blocks that fall within the (2 * BlocklistPoll) range where the value of Blocklist poll duration is defined in the storage section below.
Compactor
For more information on configuration options, see here.
Compactors stream blocks from the storage backend, combine them and write them back. Values shown below are the defaults.
Storage
For more information on configuration options, see here.
The storage block is used to configure TempoDB. It supports S3, GCS, Azure, local file system, and optionally can use Memcached or Redis for increased query performance.
The following example shows common options. For further platform-specific information refer to the following:
Memberlist
Memberlist is the default mechanism for all of the Tempo pieces to coordinate with each other.
Overrides
Tempo provides a overrides module for user to set global or per-tenant override settings. Currently only ingestion limits can be overridden.
Ingestion limits
The default limits in Tempo may not be sufficient in high volume tracing environments. Errors including RATE_LIMITED
/TRACE_TOO_LARGE
/LIVE_TRACES_EXCEEDED
will occur when these limits are exceeded.
Standard overrides
You can create an overrides
section to configure new ingestion limits that applies to all tenants of the cluster.
A snippet of a config.yaml file showing how the overrides section is here.
Tenant-specific overrides
You can set tenant-specific overrides settings in a separate file and point per_tenant_override_config
to it. This overrides file is dynamically loaded. It can be changed at runtime and will be reloaded by Tempo without restarting the application.
# /conf/tempo.yaml
# Overrides configuration block
overrides:
per_tenant_override_config: /conf/overrides.yaml
---
# /conf/overrides.yaml
# Tenant-specific overrides configuration
overrides:
"<tenant id>":
[ingestion_burst_size_bytes: <int>]
[ingestion_rate_limit_bytes: <int>]
[max_bytes_per_trace: <int>]
[max_traces_per_user: <int>]
# A "wildcard" override can be used that will apply to all tenants if a match is not found otherwise.
"*":
[ingestion_burst_size_bytes: <int>]
[ingestion_rate_limit_bytes: <int>]
[max_bytes_per_trace: <int>]
[max_traces_per_user: <int>]
Override strategies
The trace limits specified by the various parameters are, by default, applied as per-distributor limits. For example, a max_traces_per_user
setting of 10000 means that each distributor within the cluster has a limit of 10000 traces per user. This is known as a local
strategy in that the specified trace limits are local to each distributor.
A setting that applies at a local level is quite helpful in ensuring that each distributor independently can process traces up to the limit without affecting the tracing limits on other distributors.
However, as a cluster grows quite large, this can lead to quite a large quantity of traces. An alternative strategy may be to set a global
trace limit that establishes a total budget of all traces across all distributors in the cluster. The global limit is averaged across all distributors by using the distributor ring.
# /conf/tempo.yaml
overrides:
# Ingestion strategy, default is `local`.
ingestion_rate_strategy: <global|local>
Search
Tempo native search can be enabled by the following top-level setting. In microservices mode, it must be set for the distributors and queriers.
search_enabled: true
Additional search-related settings are available in the distributor and ingester sections.