Head sampling for the OpenTelemetry Collector

Overview

With a head sampling strategy, the decision to sample the trace is usually made as early as possible and does not need to take into account the whole trace - it is a simple but effective sampling strategy.

Head sampling can be implemented at the application level with SDKs, or at the collector level to keep the sampling configuration separate from the application.

Prerequisites

Head sampling should not already be implemented at the application level.
Use The OpenTelemetry Collector to collect traces from the application, generate metrics from traces, and apply sampling.
Send all traces to the collector to let the collector generate accurate metrics.

Setup

The following OpenTelemetry Collector components will be used for metrics generation and sampling:

The collector receives un-sampled traces, generates metrics, and sends metrics to Grafana Cloud Prometheus. In parallel, the collector applies a probabilistic sampling strategy to the traces and sends sampled data to Grafana Cloud Tempo.

The configuration for the collector is similar to the configuration for tail sampling strategy. The only difference is a probabilistic sampler processor is used instead of a tail sampling processor:

receivers:
  otlp:
    protocols:
      grpc:
      http:
extensions:
  basicauth/otlp:
    client_auth:
      username: <instanceID>
      password: <Cloud Access Policy token>

connectors:
  servicegraph:
    dimensions:
      - service.namespace
  spanmetrics:
    namespace: traces.spanmetrics
    histogram:
      unit: s
processors:
  memory_limiter:
  batch:
  metricstransform:
    transforms:
      - include: "traces.spanmetrics.duration"
        action: "update"
        new_name: "traces.spanmetrics.latency"
      - include: "traces.spanmetrics.calls"
        action: "update"
        new_name: "traces.spanmetrics.calls.total"
  probabilistic_sampler:
    sampling_percentage: 50
exporters:
  otlphttp:
    auth:
      authenticator: basicauth/otlp
    endpoint: https://otlp-gateway-<zone>.grafana.net/otlp

service:
  extensions: [basicauth/otlp]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [servicegraph, spanmetrics]
    traces/grafanacloud:
      receivers: [otlp]
      processors: [memory_limiter, probabilistic_sampler, batch]
      exporters: [otlphttp]
    metrics:
      receivers: [servicegraph, spanmetrics]
      processors: [memory_limiter, metricstransform, batch]
      exporters: [otlphttp]

The instrumented application sends un-sampled traces to the collector via OTLP. The collector receives data and processes it with defined pipelines.

There is one pipeline for metrics (metrics) and two pipelines for traces (traces and traces/grafanacloud).

The traces pipeline receives traces with the otlp receiver and exports them to the servicegraph and spanmetrics connectors.

The traces/grafanacloud pipeline receives traces with the otlp receiver, uses a probabilistic_sampler processor to sample traces, and exports them to the Grafana Cloud with the otlphttp exporter.

The metrics pipeline receives traces from the servicegraph and spanmetrics connectors, and applies a transform processor to align metric names produced by the spanmetrics connector with metrics produced in Tempo.

All the pipelines use batch and memory limiter processors. Batching helps to better compress the data and reduce the number of outgoing connections required to transmit the data. The memory limiter processor is used to prevent out of memory situations on the collector. It is also recommended to use the health check extension to check the status of the OpenTelemetry collector.

The otlphttp exporter is configured to export data in the OTLP format to Grafana Cloud. Consult the Grafana Cloud documentation to configure the OpenTelemetry Collector to send OTLP data to the Grafana Cloud.