The default ingestion limits in Tempo may not be sufficient in high volume tracing environments. The following message, for example, indicates that a tenant is exceeding the trace limits imposed on them.
max live traces per tenant exceeded: per-user traces limit (local: 10000 global: 0 actual local: 10000) exceeded
The following sections describe the available options, per tenant overrides and global vs. local configurations.
The following options can be used to limit ingestion:
ingestion_burst_size_bytes: Burst size (bytes) used in ingestion. Default is
ingestion_rate_limit_bytes: Per-user ingestion rate limit (bytes) used in ingestion. Default is
max_bytes_per_trace: Maximum size of a single trace in bytes.
0to disable. Default is
max_traces_per_user: Maximum number of active traces per user, per ingester.
0to disable. Default is
ingestion_rate_limit_bytes parameters control the rate limit. When these limits exceed the following message is logged:
RATE_LIMITED: ingestion rate limit (15000000 bytes) exceeded while adding 10 bytes
When the limit for the
max_bytes_per_trace parameter exceeds the following message is logged:
TRACE_TOO_LARGE: max size of trace (5000000) exceeded while adding 387 bytes
Finally, when the limit for the
max_traces_per_user parameter exceeds the following message is logged:
LIVE_TRACES_EXCEEDED: max live traces per tenant exceeded: per-user traces limit (local: 10000 global: 0 actual local: 1) exceeded
To configure new ingestion limits that applies to all tenants of the cluster:
- Create an
overridessection at the bottom of your configuration file.
- Add the parameters under this section. For example:
overrides: ingestion_burst_size_bytes: 5_000_000 ingestion_rate_limit_bytes: 5_000_000 max_bytes_per_trace: 5_000_000 max_traces_per_user: 10000
A snippet of a
config.yaml file showing how the
overrides section is here.
Sometimes you don’t want all tenants within the cluster to have the same settings. To add overrides that are specific to an individual tenant:
overridessection at the bottom of the
Add an entry
per_tenant_override_configto point to a separate file named
overrides.yamlthat contains tenant-specific overrides.
overrides: per_tenant_override_config: /conf/overrides.yaml
overrides.yamlfile, add the tenant-specific override parameters. For example:
overrides: "<tenant id>": ingestion_burst_size_bytes: 5_000_000 ingestion_rate_limit_bytes: 40_000_000 max_bytes_per_trace: 25_000_000 max_traces_per_user: 100000
This overrides file is dynamically loaded. It can be changed at runtime and will be reloaded by Tempo without restarting the application.
- Additionally a “wildcard” override can be used that will apply to all tenants if a match is not found otherwise.
overrides: "*": ingestion_burst_size_bytes: 5_000_000 ingestion_rate_limit_bytes: 40_000_000 max_bytes_per_trace: 25_000_000 max_traces_per_user: 100000
The trace limits specified by the various parameters are, by default, applied as per-distributor limits. For example, a
max_traces_per_user setting of 10000 means that each distributor within the cluster has a limit of 10000 traces per user. This is known as a
local strategy in that the specified trace limits are local to each distributor.
A setting that applies at a local level is quite helpful in ensuring that each distributor independently can process traces up to the limit without affecting the tracing limits on other distributors.
However, as a cluster grows quite large, this can lead to quite a large quantity of traces. An alternative strategy may be to set a
global trace limit that establishes a total budget of all traces across all distributors in the cluster. The global limit is averaged across all distributors by using the distributor ring.
The default setting for ingestion strategy is
local. The strategy setting can also be changed using the
overrides section, as shown in the following example:
overrides: ingestion_rate_strategy: global
Related Tempo resources
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.