Menu

This is documentation for the next version of Alloy. For the latest stable release, go to the latest version.

Documentationbreadcrumb arrow Grafana Alloybreadcrumb arrow Referencebreadcrumb arrow Componentsbreadcrumb arrow otelcolbreadcrumb arrow otelcol.processor.tail_sampling
Open source

otelcol.processor.tail_sampling

otelcol.processor.tail_sampling samples traces based on a set of defined policies. All spans for a given trace must be received by the same collector instance for effective sampling decisions.

The tail_sampling component uses both soft and hard limits, where the hard limit is always equal or larger than the soft limit. When memory usage goes above the soft limit, the processor component drops data and returns errors to the preceding components in the pipeline. When usage exceeds the hard limit, the processor forces a garbage collection in order to try and free memory. When usage is below the soft limit, no data is dropped and no forced garbage collection is performed.

Note: otelcol.processor.tail_sampling is a wrapper over the upstream OpenTelemetry Collector Contrib tail_sampling processor. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

Multiple otelcol.processor.tail_sampling components can be specified by giving them different labels.

Usage

alloy
otelcol.processor.tail_sampling "LABEL" {
  policy {
    ...
  }
  ...

  output {
    traces  = [...]
  }
}

Arguments

otelcol.processor.tail_sampling supports the following arguments:

NameTypeDescriptionDefaultRequired
decision_waitdurationWait time since the first span of a trace before making a sampling decision."30s"no
num_tracesintNumber of traces kept in memory.50000no
expected_new_traces_per_secintExpected number of new traces (helps in allocating data structures).0no
decision_cacheobjectConfigures the number of trace IDs to be kept in an LRU cache.{}no

decision_wait determines the number of batches to maintain on a channel. Its value must convert to a number of seconds greater than zero.

num_traces determines the buffer size of the trace delete channel which is composed of trace ids. Increasing the number will increase the memory usage of the component while decreasing the number will lower the maximum amount of traces kept in memory.

expected_new_traces_per_sec determines the initial slice sizing of the current batch. A larger number will use more memory but be more efficient when adding traces to the batch.

decision_cache requires a key sampled_cache_size with a value that indicates the number of trace IDs to keep in the cache. When sampled_cache_size is set to 0, the cache is inactive. When you use decision_cache, make sure you set sampled_cache_size to a value much higher than num_traces so that decisions for trace IDs are kept longer than the span data for the trace.

Blocks

The following blocks are supported inside the definition of otelcol.processor.tail_sampling:

HierarchyBlockDescriptionRequired
policypolicyPolicies used to make a sampling decision.yes
policy > latencylatencyThe policy samples based on the duration of the trace.no
policy > numeric_attributenumeric_attributeThe policy samples based on the number attributes (resource and record).no
policy > probabilisticprobabilisticThe policy samples a percentage of traces.no
policy > status_codestatus_codeThe policy samples based upon the status code.no
policy > string_attributestring_attributeThe policy samples based on string attributes (resource and record) value matches.no
policy > rate_limitingrate_limitingThe policy samples based on rate.no
policy > span_countspan_countThe policy samples based on the minimum number of spans within a batch.no
policy > boolean_attributeboolean_attributeThe policy samples based on a boolean attribute (resource and record).no
policy > ottl_conditionottl_conditionThe policy samples based on a given boolean OTTL condition (span and span event).no
policy > trace_statetrace_stateThe policy samples based on TraceState value matches.no
policy > andandThe policy samples based on multiple policies, creates an and policy.no
policy > and > and_sub_policyand_sub_policyA set of policies underneath an and policy type.no
policy > and > and_sub_policy > latencylatencyThe policy samples based on the duration of the trace.no
policy > and > and_sub_policy > numeric_attributenumeric_attributeThe policy samples based on number attributes (resource and record).no
policy > and > and_sub_policy > probabilisticprobabilisticThe policy samples a percentage of traces.no
policy > and > and_sub_policy > status_codestatus_codeThe policy samples based upon the status code.no
policy > and > and_sub_policy > string_attributestring_attributeThe policy samples based on string attributes (resource and record) value matches.no
policy > and > and_sub_policy > rate_limitingrate_limitingThe policy samples based on rate.no
policy > and > and_sub_policy > span_countspan_countThe policy samples based on the minimum number of spans within a batch.no
policy > and > and_sub_policy > boolean_attributeboolean_attributeThe policy samples based on a boolean attribute (resource and record).no
policy > and > and_sub_policy > ottl_conditionottl_conditionThe policy samples based on a given boolean OTTL condition (span and span event).no
policy > and > and_sub_policy > trace_statetrace_stateThe policy samples based on TraceState value matches.no
policy > compositecompositeThe policy samples based on a combination of above samplers, with ordering and rate allocation per sampler.no
policy > composite > composite_sub_policycomposite_sub_policyA set of policies underneath a composite policy type.no
policy > composite > composite_sub_policy > latencylatencyThe policy samples based on the duration of the trace.no
policy > composite > composite_sub_policy > numeric_attributenumeric_attributeThe policy samples based on number attributes (resource and record).no
policy > composite > composite_sub_policy > probabilisticprobabilisticThe policy samples a percentage of traces.no
policy > composite > composite_sub_policy > status_codestatus_codeThe policy samples based upon the status code.no
policy > composite > composite_sub_policy > string_attributestring_attributeThe policy samples based on string attributes (resource and record) value matches.no
policy > composite > composite_sub_policy > rate_limitingrate_limitingThe policy samples based on rate.no
policy > composite > composite_sub_policy > span_countspan_countThe policy samples based on the minimum number of spans within a batch.no
policy > composite > composite_sub_policy > boolean_attributeboolean_attributeThe policy samples based on a boolean attribute (resource and record).no
policy > composite > composite_sub_policy > ottl_conditionottl_conditionThe policy samples based on a given boolean OTTL condition (span and span event).no
policy > composite > composite_sub_policy > trace_statetrace_stateThe policy samples based on TraceState value matches.no
outputoutputConfigures where to send received telemetry data.yes
debug_metricsdebug_metricsConfigures the metrics that this component generates to monitor its state.no

policy block

The policy block configures a sampling policy used by the component. At least one policy block is required.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
namestringThe custom name given to the policy.yes
typestringThe valid policy type for this policy.yes

Each policy results in a decision, and the processor evaluates them to make a final decision:

  • When there’s an “inverted not sample” decision, the trace is not sampled.
  • When there’s a “sample” decision, the trace is sampled.
  • When there’s an “inverted sample” decision and no “not sample” decisions, the trace is sampled.
  • In all other cases, the trace is not sampled.

An “inverted” decision is the one made based on the “invert_match” attribute, such as the one from the string tag policy.

latency block

The latency block configures a policy of type latency. The policy samples based on the duration of the trace. The duration is determined by looking at the earliest start time and latest end time, without taking into consideration what happened in between.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
threshold_msnumberLower latency threshold for sampling, in milliseconds.yes
upper_threshold_msnumberUpper latency threshold for sampling, in milliseconds.0no

For a trace to be sampled, its latency should be greater than threshold_ms and lower than or equal to upper_threshold_ms.

An upper_threshold_ms of 0 will result in a policy which samples anything greater than threshold_ms.

numeric_attribute block

The numeric_attribute block configures a policy of type numeric_attribute. The policy samples based on number attributes (resource and record).

The following arguments are supported:

NameTypeDescriptionDefaultRequired
keystringTag that the filter is matched against.yes
min_valuenumberThe minimum value of the attribute to be considered a match.yes
max_valuenumberThe maximum value of the attribute to be considered a match.yes
invert_matchboolIndicates that values must not match against attribute values.falseno

probabilistic block

The probabilistic block configures a policy of type probabilistic. The policy samples a percentage of traces.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
sampling_percentagenumberThe percentage rate at which traces are sampled.yes
hash_saltstringSee below.no

Use hash_salt to configure the hashing salts. This is important in scenarios where multiple layers of collectors have different sampling rates. If multiple collectors use the same salt with different sampling rates, passing one layer may pass the other even if the collectors have different sampling rates. Configuring different salts avoids that.

status_code block

The status_code block configures a policy of type status_code. The policy samples based upon the status code.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
status_codeslist(string)Holds the configurable settings to create a status code filter sampling policy evaluator.yes

status_codes values must be “OK”, “ERROR” or “UNSET”.

string_attribute block

The string_attribute block configures a policy of type string_attribute. The policy samples based on string attributes (resource and record) value matches. Both exact and regex value matches are supported.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
keystringTag that the filter is matched against.yes
valueslist(string)Set of values or regular expressions to use when matching against attribute values.yes
enabled_regex_matchingboolDetermines whether to match attribute values by regexp string.falseno
cache_max_sizestringThe maximum number of attribute entries of Least Recently Used (LRU) Cache that stores the matched result from the regular expressions defined in values.no
invert_matchboolIndicates that values or regular expressions must not match against attribute values.falseno

rate_limiting block

The rate_limiting block configures a policy of type rate_limiting. The policy samples based on rate.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
spans_per_secondnumberSets the maximum number of spans that can be processed each second.yes

span_count block

The span_count block configures a policy of type span_count. The policy samples based on the minimum number of spans within a batch. If all traces within the batch have fewer spans than the threshold, the batch is not sampled.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
min_spansnumberMinimum number of spans in a trace.yes
max_spansnumberMaximum number of spans in a trace.0no

Set max_spans to 0, if you do not want to limit the policy samples based on the maximum number of spans in a trace.

boolean_attribute block

The boolean_attribute block configures a policy of type boolean_attribute. The policy samples based on a boolean attribute (resource and record).

The following arguments are supported:

NameTypeDescriptionDefaultRequired
keystringAttribute key to match against.yes
valueboolThe bool value (true or false) to use when matching against attribute values.yes

ottl_condition block

The ottl_condition block configures a policy of type ottl_condition. The policy samples based on a given boolean OTTL condition (span and span event).

The following arguments are supported:

NameTypeDescriptionDefaultRequired
error_modestringError handling if OTTL conditions fail to evaluate.yes
spanlist(string)OTTL conditions for spans.[]no
spaneventlist(string)OTTL conditions for span events.[]no

The supported values for error_mode are:

  • ignore: Ignore errors returned by conditions, log them, and continue on to the next condition. This is the recommended mode.
  • silent: Ignore errors returned by conditions, do not log them, and continue on to the next condition.
  • propagate: Return the error up the pipeline. This will result in the payload being dropped from Alloy.

At least one of span or spanevent should be specified. Both span and spanevent can also be specified.

trace_state block

The trace_state block configures a policy of type trace_state. The policy samples based on TraceState value matches.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
keystringTag that the filter is matched against.yes
valueslist(string)Set of values to use when matching against trace_state values.yes

and block

The and block configures a policy of type and. The policy samples based on multiple policies by creating an and policy.

and_sub_policy block

The and_sub_policy block configures a sampling policy used by the and block. At least one and_sub_policy block is required inside an and block.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
namestringThe custom name given to the policy.yes
typestringThe valid policy type for this policy.yes

composite block

The composite block configures a policy of type composite. This policy samples based on a combination of the above samplers, with ordering and rate allocation per sampler. Rate allocation allocates certain percentages of spans per policy order. For example, if max_total_spans_per_second is set to 100, then rate_allocation is set as follows:

  1. test-composite-policy-1 = 50% of max_total_spans_per_second = 50 spans_per_second
  2. test-composite-policy-2 = 25% of max_total_spans_per_second = 25 spans_per_second
  3. To ensure remaining capacity is filled, use always_sample as one of the policies.

composite_sub_policy block

The composite_sub_policy block configures a sampling policy used by the composite block. At least onecomposite_sub_policy block is required inside a composite block.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
namestringThe custom name given to the policy.yes
typestringThe valid policy type for this policy.yes

output block

The output block configures a set of components to forward resulting telemetry data to.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
logslist(otelcol.Consumer)List of consumers to send logs to.[]no
metricslist(otelcol.Consumer)List of consumers to send metrics to.[]no
traceslist(otelcol.Consumer)List of consumers to send traces to.[]no

You must specify the output block, but all its arguments are optional. By default, telemetry data is dropped. Configure the metrics, logs, and traces arguments accordingly to send telemetry data to other components.

debug_metrics block

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
disable_high_cardinality_metricsbooleanWhether to disable certain high cardinality metrics.trueno
levelstringControls the level of detail for metrics emitted by the wrapped collector."detailed"no

disable_high_cardinality_metrics is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

Note

If configured, disable_high_cardinality_metrics only applies to otelcol.exporter.* and otelcol.receiver.* components.

level is the Alloy equivalent to the telemetry.metrics.level feature gate in the OpenTelemetry Collector. Possible values are "none", "basic", "normal" and "detailed".

Exported fields

The following fields are exported and can be referenced by other components:

NameTypeDescription
inputotelcol.ConsumerA value that other components can use to send telemetry data to.

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.processor.tail_sampling is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.processor.tail_sampling does not expose any component-specific debug information.

Example

This example batches trace data from Alloy before sending it to otelcol.exporter.otlp for further processing. This example shows an impractical number of policies for the purpose of demonstrating how to set up each type.

alloy
tracing {
  sampling_fraction = 1
  write_to          = [otelcol.processor.tail_sampling.default.input]
}

otelcol.processor.tail_sampling "default" {
  decision_cache = {
    sampled_cache_size = 100000,
    }
  decision_wait               = "10s"
  num_traces                  = 100
  expected_new_traces_per_sec = 10

  policy {
    name = "test-policy-1"
    type = "always_sample"
  }

  policy {
    name = "test-policy-2"
    type = "latency"

    latency {
      threshold_ms = 5000
    }
  }

  policy {
    name = "test-policy-3"
    type = "numeric_attribute"

    numeric_attribute {
      key       = "key1"
      min_value = 50
      max_value = 100
    }
  }

  policy {
    name = "test-policy-4"
    type = "probabilistic"

    probabilistic {
      sampling_percentage = 10
    }
  }

  policy {
    name = "test-policy-5"
    type = "status_code"

    status_code {
      status_codes = ["ERROR", "UNSET"]
    }
  }

  policy {
    name = "test-policy-6"
    type = "string_attribute"

    string_attribute {
      key    = "key2"
      values = ["value1", "value2"]
    }
  }

  policy {
    name = "test-policy-7"
    type = "string_attribute"

    string_attribute {
      key                    = "key2"
      values                 = ["value1", "val*"]
      enabled_regex_matching = true
      cache_max_size         = 10
    }
  }

  policy {
    name = "test-policy-8"
    type = "rate_limiting"

    rate_limiting {
      spans_per_second = 35
    }
  }

  policy {
    name = "test-policy-9"
    type = "string_attribute"

    string_attribute {
      key                    = "http.url"
      values                 = ["/health", "/metrics"]
      enabled_regex_matching = true
      invert_match           = true
    }
  }

  policy {
    name = "test-policy-10"
    type = "span_count"

    span_count {
      min_spans = 2
    }
  }

  policy {
    name = "test-policy-11"
    type = "trace_state"

    trace_state {
      key    = "key3"
      values = ["value1", "value2"]
    }
  }

  policy {
    name = "test-policy-12"
    type = "ottl_condition"
    ottl_condition {
      error_mode = "ignore"
      span = [
        "attributes[\"test_attr_key_1\"] == \"test_attr_val_1\"",
        "attributes[\"test_attr_key_2\"] != \"test_attr_val_1\"",
      ]
      spanevent = [
        "name != \"test_span_event_name\"",
        "attributes[\"test_event_attr_key_2\"] != \"test_event_attr_val_1\"",
      ]
    }
  }

  policy {
    name = "and-policy-1"
    type = "and"

    and {
      and_sub_policy {
        name = "test-and-policy-1"
        type = "numeric_attribute"

        numeric_attribute {
          key       = "key1"
          min_value = 50
          max_value = 100
        }
      }

      and_sub_policy {
        name = "test-and-policy-2"
        type = "string_attribute"

        string_attribute {
          key    = "key1"
          values = ["value1", "value2"]
        }
      }
    }
  }

  policy {
    name = "composite-policy-1"
    type = "composite"

    composite {
      max_total_spans_per_second = 1000
      policy_order               = ["test-composite-policy-1", "test-composite-policy-2", "test-composite-policy-3"]

      composite_sub_policy {
        name = "test-composite-policy-1"
        type = "numeric_attribute"

        numeric_attribute {
          key       = "key1"
          min_value = 50
          max_value = 100
        }
      }

      composite_sub_policy {
        name = "test-composite-policy-2"
        type = "string_attribute"

        string_attribute {
          key    = "key1"
          values = ["value1", "value2"]
        }
      }

      composite_sub_policy {
        name = "test-composite-policy-3"
        type = "always_sample"
      }

      rate_allocation {
        policy  = "test-composite-policy-1"
        percent = 50
      }

      rate_allocation {
        policy  = "test-composite-policy-2"
        percent = 50
      }
    }
  }

  output {
    traces = [otelcol.exporter.otlp.production.input]
  }
}

otelcol.exporter.otlp "production" {
  client {
    endpoint = sys.env("OTLP_SERVER_ENDPOINT")
  }
}

Compatible components

otelcol.processor.tail_sampling can accept arguments from the following components:

otelcol.processor.tail_sampling has exports that can be consumed by the following components:

Note

Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.