Tail-based sampling
Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.
Tempo aims to provide an inexpensive solution that makes 100% sampling possible. However, sometimes constraints make a lower sampling percentage necessary or desirable, such as runtime or egress traffic related costs. Probabilistic sampling strategies are easy to implement, but also run the risk of discarding relevant data that you’ll later want.
Tail-based sampling works with Grafana Agent in Flow or static modes. Flow mode configuration files are written in River. Static mode configuration files are written in YAML. Examples in this document are for Flow mode. You can also use the Static mode Kubernetes operator.
How tail-based sampling works
In tail-based sampling, sampling decisions are made at the end of the workflow allowing for a more accurate sampling decision.
The Grafana Agent groups spans by trace ID and checks its data to see
if it meets one of the defined policies (for example, latency
or status_code
).
For instance, a policy can check if a trace contains an error or if it took
longer than a certain duration.
A trace is sampled if it meets at least one policy.
To group spans by trace ID, the Agent buffers spans for a configurable amount of time, after which it considers the trace complete. Longer running traces are split into more than one. However, waiting longer times increases the memory overhead of buffering.
One particular challenge of grouping trace data is for multi-instance Agent deployments, where spans that belong to the same trace can arrive to different Agents. To solve that, you can configure the Agent to load balance traces across agent instances by exporting spans belonging to the same trace to the same instance.
This is achieved by redistributing spans by trace ID once they arrive from the application. The Agent must be able to discover and connect to other Agent instances where spans for the same trace can arrive. Kubernetes users should use a headless service.
Redistributing spans by trace ID means that spans are sent and received twice, which can cause a significant increase in CPU usage. This overhead increases with the number of Agent instances that share the same traces.
Quickstart
To start using tail-based sampling, define a sampling policy. If you’re using a multi-instance deployment of the agent, add load balancing and specify the resolving mechanism to find other Agents in the setup. To see all the available configuration options, refer to the configuration reference.
Note
Grafana Alloy provides tooling to convert your Agent Static or Flow configuration files into a format that can be used by Alloy.
For more information, refer to Migrate to Alloy.
Example for Grafana Agent Flow
Grafana Agent Flow is a component-based revision of Grafana Agent with a focus on ease-of-use, debuggability, and ability to adapt to the needs of power users. Flow configuration files are written in River instead of YAML.
Grafana Agent Flow uses the otelcol.processor.tail_sampling component
` for tail-based sampling.
Examples for Grafana Agent static mode
For additional information, refer to the blog post, An introduction to trace sampling with Grafana Tempo and Grafana Agent.
Status code tail sampling policy
The following policy only samples traces where at least one span contains an OpenTelemetry Error status code.
traces:
configs:
- name: default
...
tail_sampling:
policies:
- type: status_code
status_code:
status_codes:
- ERROR
Span attribute tail sampling policy
The following policy only samples traces where the span attribute http.target
does not contain the value /healthcheck
or is prefixed with /metrics/
.
traces:
configs:
- name: default
tail_sampling:
policies:
- type: string_attribute
string_attribute:
key: http.target
values:
- ^\/(?:metrics\/.*|healthcheck)$
enabled_regex_matching: true
invert_match: true
And compound tail sampling policy
The following policy will only sample traces where all of the conditions for the sub-policies are met.
In this case, it takes the prior two policies and will only sample traces where the span attribute http.target
does not contain the value /healthcheck
or is prefixed with /metrics/
and at least one of the spans of the trace contains an OpenTelemetry Error status code.
traces:
configs:
- name: default
tail_sampling:
policies:
- type: and
and_sub_policy:
- name: and_tag_policy
type: string_attribute
string_attribute:
key: http.target
values:
- ^\/(?:metrics\/.*|healthcheck)$
enabled_regex_matching: true
invert_match: true
- name: and_error_policy
type: status_code
status_code:
status_codes:
- ERROR