Tail sampling for the OpenTelemetry Collector

Overview

With a tail sampling strategy, the decision to sample the trace is made considering all or most of the spans. For example, tail sampling is a good option when you want to sample only traces that have errors or traces with long request duration.

Prerequisites

Head sampling should not already be implemented at the application level.
Use The OpenTelemetry Collector to collect traces from the application, generate metrics from traces, and apply sampling.
Send all traces to the collector to let the collector generate accurate metrics.

All the traces need to be sent because not all SDKs set the required trace state values.

Setup

The following OpenTelemetry collector components will be used for metrics generation and sampling:

The collector receives un-sampled traces, generates metrics, and sends metrics to Grafana Cloud Prometheus. In parallel, the collector applies a tail sampling strategy to the traces and sends sampled data to Grafana Cloud Tempo.

The configuration for the collector is similar to the configuration for head sampling strategy. The only difference is a tail sampling processor is used instead of a probabilistic sampler processor:

receivers:
  otlp:
    protocols:
      grpc:
      http:
extensions:
  basicauth/otlp:
    client_auth:
      username: <instanceID>
      password: <Cloud Access Policy token>

connectors:
  servicegraph:
    dimensions:
      - service.namespace
  spanmetrics:
    namespace: traces.spanmetrics
    histogram:
      unit: s
processors:
  memory_limiter:
  batch:
  metricstransform:
    transforms:
      - include: "traces.spanmetrics.duration"
        action: "update"
        new_name: "traces.spanmetrics.latency"
      - include: "traces.spanmetrics.calls"
        action: "update"
        new_name: "traces.spanmetrics.calls.total"
  tail_sampling:
    policies:
      [{ name: latency, type: latency, latency: { threshold_ms: 5000 } }]
exporters:
  otlphttp:
    auth:
      authenticator: basicauth/otlp
    endpoint: https://otlp-gateway-<zone>.grafana.net/otlp

service:
  extensions: [basicauth/otlp]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [servicegraph, spanmetrics]
    traces/grafanacloud:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlphttp]
    metrics:
      receivers: [servicegraph, spanmetrics]
      processors: [memory_limiter, metricstransform, batch]
      exporters: [otlphttp]

The instrumented application sends un-sampled traces to the collector via OTLP. The collector receives data and processes it with defined pipelines.

There is one pipeline for metrics (metrics) and two pipelines for traces (traces and traces/grafanacloud).

The traces pipeline receives traces with the otlp receiver and exports them to the servicegraph and spanmetrics connectors.

The traces/grafanacloud pipeline receives traces with the otlp receiver, uses a tail sampling processor to sample traces, and exports them to the Grafana Cloud with the otlphttp exporter.

The default configuration of the tail sampling processor will hold traces in memory for 30s before sampling. Once a decision to sample has been made, spans that arrive after will be treated as new traces.

Consult the tail sampling processor README.md for a list of configuration options.

The metrics pipeline receives traces from the the servicegraph and spanmetrics connectors, and applies a transform processor to align metric names produced by the spanmetrics connector with metrics produced in Tempo.

All the pipelines use batch and memory limiter processors. Batching helps to better compress the data and reduce the number of outgoing connections required to transmit the data. The memory limiter processor is used to prevent out of memory situations on the collector. It is also recommended to use the health check extension to check the status of the OpenTelemetry Collector.

The otlphttp exporter is configured to export data in OTLP format to Grafana Cloud. Consult the Grafana Cloud documentation to configure the OpenTelemetry Collector to send OTLP data to the Grafana Cloud.

Scaling

Scaling a tail sampling setup requires some thought and more planning, and typically involves the use of a load-balancing exporter.

Scaling the load-balancing exporter is easy, as an off-the-shelf layer 4 load-balancer would be sufficient.

With a Kubernetes setup, the OpenTelemetry Collector configuration for the load-balancer exporter could look like this:

receivers:
  otlp:
    protocols:
      grpc:

processors:

exporters:
  loadbalancing:
    protocol:
      otlp:
    resolver:
      static:
        hostnames:
          - otel-collector-1:4317
          - otel-collector-2:4317
          - otel-collector-3:4317

service:
  pipelines:
    traces:
      receivers:
        - otlp
      exporters:
        - loadbalancing

The collector has three resolvers for the load-balancing exporter static, dns, and k8s.

static: A static list of backends is provided in the configuration. This is suitable when the backends are static and scaling isn’t expected.
dns: A hostname is provided as a parameter which the resolver periodically queries to discover IPs and update the load-balancer ring. When multiple instances are used, there is a chance they can momentarily have a different view of the system while they sync after a refresh. This can result in some spans for the same trace ID being sent to multiple hosts. Determine if this acceptable for the system, and use a longer refresh interval to reduce the effect of being out of sync.
k8s: A new resolver that implements a watcher using Kubernetes APIs to get notifications when the list of pods backing a service is changed. This should reduce the amount of time when cluster views differ between nodes, effectively being a better solution than the DNS resolver when Kubernetes is used.

The static and dns resolvers are recommended for use in production environments. The k8s resolver is new and experimental.