Documentation Grafana Cloud Observe and act Optimize data Adaptive Traces Guides Optimize egress costs

Optimize egress costs with multi-tier sampling

By default, you send 100% of your spans to Grafana Cloud, and Adaptive Traces makes the final keep-or-drop decision for every trace there. Sending everything gives Adaptive Traces the most context to work with, so policies like diversity, latency, and errors can pick out the rare, valuable traces that a random sampling strategy would miss. The tradeoff is cost, because you pay to transmit and process every span before Adaptive Traces decides what to keep.

Multi-tier sampling lowers that cost by adding a second sampling tier closer to your applications, in Grafana Alloy or the OpenTelemetry Collector. This tier samples traffic before it leaves your environment, which reduces egress and processing costs, especially at large scale. Adaptive Traces still makes the final decision about what is stored in Tempo. Lowering the collector rate saves more on egress, but gives Adaptive Traces fewer traces to choose from, so some important traces may be missed.

Historically, sampling before Grafana Cloud could cause issues, as the tail sampling and metrics generation pipelines would no longer see the full stream to know how much traffic was removed. The OpenTelemetry trace state solves this. It’s a small piece of sampling metadata that travels with each span and records how aggressively the collector sampled. This guide explains how the two tiers fit together, how to configure the collector tier so the trace state is recorded, and how the collector sampling rate relates to the sampling rate in your Adaptive Traces policies.

The two tiers

The following diagram shows how traffic flows through the two tiers.

flowchart LR
    subgraph env["Your environment"]
        app["Your applications"]
        coll["Collector tier: Alloy or OpenTelemetry Collector"]
    end
    subgraph cloud["Grafana Cloud"]
        at["Adaptive Traces tail sampling"]
        tempo["Tempo"]
    end

    app -->|"100% of spans"| coll
    coll -->|"sampled subset with sampling probability in trace state"| at
    at -->|"traces worth keeping"| tempo

The collector tier only controls how much data reaches Grafana Cloud. It doesn’t make keep-or-drop decisions about individual traces. Adaptive Traces still makes the final decision about what is stored from whatever arrives.

Configure the collector tier

Use the probabilistic sampler in either Alloy or the OpenTelemetry Collector. It records, in the trace state, the probability that a trace was kept, allowing tail sampling decisions to remain consistent and metrics to be accurately extrapolated.

Grafana Alloy

The following pipeline samples 25% of traffic and forwards the rest to Grafana Cloud.

Alloy Copy

otelcol.receiver.otlp "default" {
  grpc {}
  http {}

  output {
    traces = [otelcol.processor.probabilistic_sampler.default.input]
  }
}

otelcol.processor.probabilistic_sampler "default" {
  sampling_percentage = 25

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    traces = [otelcol.exporter.otlp.grafana_cloud.input]
  }
}

otelcol.exporter.otlp "grafana_cloud" {
  client {
    endpoint = "<GRAFANA_CLOUD_OTLP_ENDPOINT>"
    auth     = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.auth.basic "grafana_cloud" {
  username = "<GRAFANA_CLOUD_INSTANCE_ID>"
  password = "<GRAFANA_CLOUD_API_TOKEN>"
}

For all available arguments, refer to otelcol.processor.probabilistic_sampler.

OpenTelemetry Collector

The equivalent configuration for the OpenTelemetry Collector:

YAML Copy

processors:
  probabilistic_sampler:
    sampling_percentage: 25

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, batch]
      exporters: [otlp]

For all available options, refer to the upstream probabilisticsamplerprocessor documentation.

Set your Adaptive Traces tail policies

Configure your Adaptive Traces policies as you normally would.

The sampling percentage you set in any policy applies to your original, pre-sampled traffic, not to what arrives after collector sampling. The tail sampler reads the trace state and adjusts the rate it applies locally to reach that target. You set the target against total traffic, and Adaptive Traces applies the matching local rate.

How the two rates interact

You can move the collector rate and the tail sampling rate independently, because both are measured against the same baseline of 100% original traffic. That way collector rates can change based on egress budget without a simultaneous change to each Adaptive Traces policy.

• The collector rate controls egress and processing cost. It determines how much data leaves your environment.
• The tail sampling rate controls storage cost. It’s the absolute target for how much of the original traffic to store in Tempo.

Because the tail rate is always a fraction of the original traffic, changing one rate never forces you to recompute the other.

Worked example

Start with 1,000 traces per second of original traffic. The collector keeps 40%, and your Adaptive Traces policy keeps 10% of the original traffic.

Now suppose you halve the collector rate to 20% of original to save more on egress. You change nothing in Adaptive Traces.

The tail sampling target never changed. It’s still “10% of original traffic.” The tail sampler discovered the new upstream rate from the trace state and adjusted its local rate from 25% to 50% automatically, so it still lands on the same 100 traces per second.

The same is true in reverse. Raise or lower your tail sampling policy and the collector rate is untouched.

In both tables, the collector output (400, then 200 traces per second) is what leaves your environment, so it drives your egress and processing cost. The tail sampling output (100 traces per second) is what Adaptive Traces stores in Tempo, so it drives your storage cost.

Choose a collector sampling rate

Start from your cost goal. Reducing the egress and processing cost of your traces is the reason to add the collector tier, so choose a rate that removes enough volume to reach the cost reduction you’re targeting.

Then consider the effect a low rate has on fidelity. The collector’s probabilistic sampler decides based only on the trace ID, so it can’t tell errors, high-latency traces, or specific attributes apart from ordinary traffic. Whatever it drops is gone before your Adaptive Traces policies see it. The lower the collector rate, the more of your errors and high-latency traces are dropped this way, so your policies capture them less often. For example, if a policy keeps 100% of errors but the collector already dropped 70% of all traffic, you still see only about 30% of your errors.

This effect is most notable for lower traffic environments where interesting events may not be sent to Grafana Cloud at all. As your scale increases, the collector tier sampling rate can typically also be lowered, as there is more data and a lower percentage still results in a representative sample. Leaving headroom above your storage goal follows the same reasoning as the best practice against using drop policies for rate limiting: sample rather than drop, so your anomaly and diversity policies still have traces to work with.

Keep two rules in mind:

• Don’t set the collector rate below your total tail target. If the collector keeps less of the original traffic than your policies want, the tail sampler can only keep what arrives, and you fall short of your target.
• Start high, then lower the rate gradually. Begin with more headroom than you expect to need, then reduce the collector rate toward your cost goal while you confirm that errors and other important traces are still captured. 50% is a good starting point that balances egress and processing savings with tail sampling effectiveness. Because the tail policy is absolute, you can change the collector rate without editing your policies.

Keep span metrics accurate

Sampling shouldn’t distort the metrics you derive from your traces. Because sampling now happens in your environment, Grafana Cloud no longer receives every trace and can’t build metrics by counting the traces it sees. Instead, it extrapolates back to the original population using the sampling probability recorded in each trace’s trace state.

Each kept trace carries an adjusted count, which is the inverse of the probability with which it was kept. If the collector kept 20% of traffic, each kept trace has an adjusted count of 1 / 0.20 = 5 and counts as 5 toward the request, error, and duration metrics. Adding up adjusted counts, rather than counting raw traces, reconstructs the metrics for 100% of the original traffic even though only a fraction of traces reach Grafana Cloud. You lose stored example traces, but not metric accuracy, so your span metric dashboards and alerts stay accurate even when you sample aggressively at the collector.

Extrapolate in TraceQL metrics queries

The extrapolation above applies to the span metrics that Grafana Cloud generates for you. When you run TraceQL metrics queries yourself, they compute metrics on demand from the sampled traces stored in Tempo, so their results reflect only the sampled traffic. Add the experimental with(extrapolate=true) query hint to scale the result back to the original traffic. Tempo reads the sampling probability from each span’s trace state and scales that span’s contribution by 1 / sampling_probability.

traceql Copy

{ resource.service.name="api" } | rate() with(extrapolate=true)

The hint applies to rate, count_over_time, sum_over_time, avg_over_time, histogram_over_time, quantile_over_time, and compare. It doesn’t change min_over_time or max_over_time, because extremes don’t scale with sampling. For more detail, refer to the Tempo metrics functions documentation.

Summary

Multi-tier sampling reduces the egress and processing cost of your traces in your own environment while Adaptive Traces keeps making the final decision in Grafana Cloud. The OpenTelemetry trace state carries the sampling probability the collector applied, which keeps the two tiers consistent, keeps tail sampling rates absolute against your original traffic, and lets Grafana Cloud extrapolate span metrics back to 100%.