This is documentation for the next version of Alloy. For the latest stable release, go to the latest version.
otelcol.processor.tail_sampling
otelcol.processor.tail_sampling
samples traces based on a set of defined
policies. All spans for a given trace must be received by the same collector
instance for effective sampling decisions.
Note
otelcol.processor.tail_sampling
is a wrapper over the upstream OpenTelemetry Collector Contribtail_sampling
processor. Bug reports or feature requests will be redirected to the upstream repository, if necessary.
Multiple otelcol.processor.tail_sampling
components can be specified by
giving them different labels.
Usage
otelcol.processor.tail_sampling "LABEL" {
policy {
...
}
...
output {
traces = [...]
}
}
Arguments
otelcol.processor.tail_sampling
supports the following arguments:
decision_wait
determines the number of batches to maintain on a channel. Its value must convert to a number of seconds greater than zero.
num_traces
determines the buffer size of the trace delete channel which is composed of trace ids. Increasing the number will increase the memory usage of the component while decreasing the number will lower the maximum amount of traces kept in memory.
expected_new_traces_per_sec
determines the initial slice sizing of the current batch. A larger number will use more memory but be more efficient when adding traces to the batch.
decision_cache
can contain two keys:
sampled_cache_size
: Configures the number of trace IDs to be kept in an LRU cache, persisting the “keep” decisions for traces that may have already been released from memory. By default, the size is 0 and the cache is inactive.non_sampled_cache_size
: Configures number of trace IDs to be kept in an LRU cache, persisting the “drop” decisions for traces that may have already been released from memory. By default, the size is 0 and the cache is inactive.
You may want to vary the size of the decision_cache
depending on how many “keep” vs “drop” decisions you expect from your policies.
For example, you can allocate a larger non_sampled_cache_size
if you expect most traces to be dropped.
Additionally, when you use decision_cache
, configure it with a much higher value than num_traces
so decisions for trace IDs are kept longer than the span data for the trace.
Blocks
The following blocks are supported inside the definition of
otelcol.processor.tail_sampling
:
policy block
The policy
block configures a sampling policy used by the component. At least one policy
block is required.
The following arguments are supported:
Each policy results in a decision, and the processor evaluates them to make a final decision:
- When there’s an “inverted not sample” decision, the trace is not sampled.
- When there’s a “sample” decision, the trace is sampled.
- When there’s an “inverted sample” decision and no “not sample” decisions, the trace is sampled.
- In all other cases, the trace is not sampled.
An “inverted” decision is the one made based on the “invert_match” attribute, such as the one from the string, numeric or boolean tag policy.
latency block
The latency
block configures a policy of type latency
. The policy samples based on the duration of the trace. The duration is determined by looking at the earliest start time and latest end time, without taking into consideration what happened in between.
The following arguments are supported:
For a trace to be sampled, its latency should be greater than threshold_ms
and lower than or equal to upper_threshold_ms
.
An upper_threshold_ms
of 0
will result in a policy which samples anything greater than threshold_ms
.
numeric_attribute block
The numeric_attribute
block configures a policy of type numeric_attribute
. The policy samples based on number attributes (resource and record).
The following arguments are supported:
probabilistic block
The probabilistic
block configures a policy of type probabilistic
. The policy samples a percentage of traces.
The following arguments are supported:
Use hash_salt
to configure the hashing salts. This is important in scenarios where multiple layers of collectors
have different sampling rates. If multiple collectors use the same salt with different sampling rates, passing one
layer may pass the other even if the collectors have different sampling rates. Configuring different salts avoids that.
status_code block
The status_code
block configures a policy of type status_code
. The policy samples based upon the status code.
The following arguments are supported:
status_codes
values must be “OK”, “ERROR” or “UNSET”.
string_attribute block
The string_attribute
block configures a policy of type string_attribute
. The policy samples based on string attributes (resource and record) value matches. Both exact and regex value matches are supported.
The following arguments are supported:
rate_limiting block
The rate_limiting
block configures a policy of type rate_limiting
. The policy samples based on rate.
The following arguments are supported:
span_count block
The span_count
block configures a policy of type span_count
. The policy samples based on the minimum number of spans within a batch. If all traces within the batch have fewer spans than the threshold, the batch is not sampled.
The following arguments are supported:
Set max_spans
to 0
, if you do not want to limit the policy samples based on the maximum number of spans in a trace.
boolean_attribute block
The boolean_attribute
block configures a policy of type boolean_attribute
.
The policy samples based on a boolean attribute (resource and record).
The following arguments are supported:
ottl_condition block
The ottl_condition
block configures a policy of type ottl_condition
. The policy samples based on a given boolean
OTTL condition (span and span event).
The following arguments are supported:
The supported values for error_mode
are:
ignore
: Ignore errors returned by conditions, log them, and continue on to the next condition. This is the recommended mode.silent
: Ignore errors returned by conditions, do not log them, and continue on to the next condition.propagate
: Return the error up the pipeline. This will result in the payload being dropped from Alloy.
At least one of span
or spanevent
should be specified. Both span
and spanevent
can also be specified.
trace_state block
The trace_state
block configures a policy of type trace_state
. The policy samples based on TraceState value matches.
The following arguments are supported:
and block
The and
block configures a policy of type and
. The policy samples based on multiple policies by creating an and
policy.
and_sub_policy block
The and_sub_policy
block configures a sampling policy used by the and
block. At least one and_sub_policy
block is required inside an and
block.
The following arguments are supported:
composite block
The composite
block configures a policy of type composite
. This policy samples based on a combination of the above samplers, with ordering and rate allocation per sampler. Rate allocation allocates certain percentages of spans per policy order. For example, if max_total_spans_per_second
is set to 100, then rate_allocation
is set as follows:
- test-composite-policy-1 = 50% of max_total_spans_per_second = 50 spans_per_second
- test-composite-policy-2 = 25% of max_total_spans_per_second = 25 spans_per_second
- To ensure remaining capacity is filled, use always_sample as one of the policies.
composite_sub_policy block
The composite_sub_policy
block configures a sampling policy used by the composite
block. At least onecomposite_sub_policy
block is required inside a composite
block.
The following arguments are supported:
output block
The output
block configures a set of components to forward resulting telemetry data to.
The following arguments are supported:
You must specify the output
block, but all its arguments are optional.
By default, telemetry data is dropped.
Configure the metrics
, logs
, and traces
arguments accordingly to send telemetry data to other components.
debug_metrics block
The debug_metrics
block configures the metrics that this component generates to monitor its state.
The following arguments are supported:
disable_high_cardinality_metrics
is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics
feature gate in the OpenTelemetry Collector.
It removes attributes that could cause high cardinality metrics.
For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.
Note
If configured,disable_high_cardinality_metrics
only applies tootelcol.exporter.*
andotelcol.receiver.*
components.
level
is the Alloy equivalent to the telemetry.metrics.level
feature gate in the OpenTelemetry Collector.
Possible values are "none"
, "basic"
, "normal"
and "detailed"
.
Exported fields
The following fields are exported and can be referenced by other components:
input
accepts otelcol.Consumer
data for any telemetry signal (metrics,
logs, or traces).
Component health
otelcol.processor.tail_sampling
is only reported as unhealthy if given an invalid
configuration.
Debug information
otelcol.processor.tail_sampling
does not expose any component-specific debug
information.
Example
This example batches trace data from Alloy before sending it to otelcol.exporter.otlp for further processing. This example shows an impractical number of policies for the purpose of demonstrating how to set up each type.
Compatible components
otelcol.processor.tail_sampling
can accept arguments from the following components:
- Components that export OpenTelemetry
otelcol.Consumer
otelcol.processor.tail_sampling
has exports that can be consumed by the following components:
- Components that consume OpenTelemetry
otelcol.Consumer
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.