---
title: "Optimize egress costs with multi-tier sampling | Grafana Cloud documentation"
description: "Learn how to combine collector-side probabilistic sampling with Adaptive Traces tail sampling using OpenTelemetry trace state, so you can reduce the egress and processing cost of your traces without distorting span metrics."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Optimize egress costs with multi-tier sampling

By default, you send 100% of your spans to Grafana Cloud, and Adaptive Traces makes the final keep-or-drop decision for every trace there. Sending everything gives Adaptive Traces the most context to work with, so policies like diversity, latency, and errors can pick out the rare, valuable traces that a random sampling strategy would miss. The tradeoff is cost, because you pay to transmit and process every span before Adaptive Traces decides what to keep.

Multi-tier sampling lowers that cost by adding a second sampling tier closer to your applications, in Grafana Alloy or the OpenTelemetry Collector. This tier samples traffic before it leaves your environment, which reduces egress and processing costs, especially at large scale. Adaptive Traces still makes the final decision about what is stored in Tempo. Lowering the collector rate saves more on egress, but gives Adaptive Traces fewer traces to choose from, so some important traces may be missed.

Historically, sampling before Grafana Cloud could cause issues, as the tail sampling and metrics generation pipelines would no longer see the full stream to know how much traffic was removed. The OpenTelemetry [trace state](https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/) solves this. It’s a small piece of sampling metadata that travels with each span and records how aggressively the collector sampled. This guide explains how the two tiers fit together, how to configure the collector tier so the trace state is recorded, and how the collector sampling rate relates to the sampling rate in your Adaptive Traces policies.

## The two tiers

The following diagram shows how traffic flows through the two tiers.

```
flowchart LR
    subgraph env["Your environment"]
        app["Your applications"]
        coll["Collector tier: Alloy or OpenTelemetry Collector"]
    end
    subgraph cloud["Grafana Cloud"]
        at["Adaptive Traces tail sampling"]
        tempo["Tempo"]
    end

    app -->|"100% of spans"| coll
    coll -->|"sampled subset with sampling probability in trace state"| at
    at -->|"traces worth keeping"| tempo
```

Expand table

| Tier               | Where it runs                                             | What it does                                                                                                                                                        |
|--------------------|-----------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Collector sampling | Alloy or the OpenTelemetry Collector, in your environment | Applies consistent probabilistic sampling to control how much traffic reaches Grafana Cloud.                                                                        |
| Tail sampling      | Adaptive Traces, in Grafana Cloud                         | Waits for the complete trace and applies your policies to make the final keep-or-drop decision. Reads the trace state to understand what the collector already did. |

The collector tier only controls how much data reaches Grafana Cloud. It doesn’t make keep-or-drop decisions about individual traces. Adaptive Traces still makes the final decision about what is stored from whatever arrives.

## Configure the collector tier

Use the probabilistic sampler in either Alloy or the OpenTelemetry Collector. It records, in the trace state, the probability that a trace was kept, allowing tail sampling decisions to remain consistent and metrics to be accurately extrapolated.

### Grafana Alloy

The following pipeline samples 25% of traffic and forwards the rest to Grafana Cloud.

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
otelcol.receiver.otlp "default" {
  grpc {}
  http {}

  output {
    traces = [otelcol.processor.probabilistic_sampler.default.input]
  }
}

otelcol.processor.probabilistic_sampler "default" {
  sampling_percentage = 25

  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    traces = [otelcol.exporter.otlp.grafana_cloud.input]
  }
}

otelcol.exporter.otlp "grafana_cloud" {
  client {
    endpoint = "<GRAFANA_CLOUD_OTLP_ENDPOINT>"
    auth     = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.auth.basic "grafana_cloud" {
  username = "<GRAFANA_CLOUD_INSTANCE_ID>"
  password = "<GRAFANA_CLOUD_API_TOKEN>"
}
```

For all available arguments, refer to [`otelcol.processor.probabilistic_sampler`](/docs/grafana-cloud/observe-and-act/send-data/alloy/reference/components/otelcol/otelcol.processor.probabilistic_sampler/).

### OpenTelemetry Collector

The equivalent configuration for the OpenTelemetry Collector:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
processors:
  probabilistic_sampler:
    sampling_percentage: 25

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, batch]
      exporters: [otlp]
```

For all available options, refer to the upstream [`probabilisticsamplerprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/README.md) documentation.

## Set your Adaptive Traces tail policies

Configure your Adaptive Traces policies as you normally would.

The sampling percentage you set in any policy applies to your **original, pre-sampled traffic**, not to what arrives after collector sampling. The tail sampler reads the trace state and adjusts the rate it applies locally to reach that target. You set the target against total traffic, and Adaptive Traces applies the matching local rate.

## How the two rates interact

You can move the collector rate and the tail sampling rate independently, because both are measured against the same baseline of 100% original traffic. That way collector rates can change based on egress budget without a simultaneous change to each Adaptive Traces policy.

- The **collector rate** controls egress and processing cost. It determines how much data leaves your environment.
- The **tail sampling rate** controls storage cost. It’s the absolute target for how much of the original traffic to store in Tempo.

Because the tail rate is always a fraction of the original traffic, changing one rate never forces you to recompute the other.

### Worked example

Start with 1,000 traces per second of original traffic. The collector keeps 40%, and your Adaptive Traces policy keeps 10% of the original traffic.

Expand table

| Tier          | Target (of original) | Rate it applies locally               | Traces per second out |
|---------------|----------------------|---------------------------------------|-----------------------|
| Collector     | keep 40%             | 40% of its input                      | 400                   |
| Tail sampling | keep 10%             | `10% / 40% = 25%` of what it receives | 100                   |

Now suppose you halve the collector rate to 20% of original to save more on egress. You change nothing in Adaptive Traces.

Expand table

| Tier          | Target (of original) | Rate it applies locally               | Traces per second out |
|---------------|----------------------|---------------------------------------|-----------------------|
| Collector     | keep 20%             | 20% of its input                      | 200                   |
| Tail sampling | **still keep 10%**   | `10% / 20% = 50%` of what it receives | 100                   |

The tail sampling target never changed. It’s still “10% of original traffic.” The tail sampler discovered the new upstream rate from the trace state and adjusted its local rate from 25% to 50% automatically, so it still lands on the same 100 traces per second.

The same is true in reverse. Raise or lower your tail sampling policy and the collector rate is untouched.

In both tables, the collector output (400, then 200 traces per second) is what leaves your environment, so it drives your egress and processing cost. The tail sampling output (100 traces per second) is what Adaptive Traces stores in Tempo, so it drives your storage cost.

> Note
> 
> Because targets are absolute rather than multiplicative, a collector tier and a tail tier that each “sample 10%” do **not** combine to 1%. Both are measured against the original population, so the effective result is 10% of original traffic, and the collector rate only controls how much traffic reaches tail sampling.

## Choose a collector sampling rate

Start from your cost goal. Reducing the egress and processing cost of your traces is the reason to add the collector tier, so choose a rate that removes enough volume to reach the cost reduction you’re targeting.

Then consider the effect a low rate has on fidelity. The collector’s probabilistic sampler decides based only on the trace ID, so it can’t tell errors, high-latency traces, or specific attributes apart from ordinary traffic. Whatever it drops is gone before your Adaptive Traces policies see it. The lower the collector rate, the more of your errors and high-latency traces are dropped this way, so your policies capture them less often. For example, if a policy keeps 100% of errors but the collector already dropped 70% of all traffic, you still see only about 30% of your errors.

This effect is most notable for lower traffic environments where interesting events may not be sent to Grafana Cloud at all. As your scale increases, the collector tier sampling rate can typically also be lowered, as there is more data and a lower percentage still results in a representative sample. Leaving headroom above your storage goal follows the same reasoning as the [best practice against using drop policies for rate limiting](../best-practices-policies/#create-drop-policies): sample rather than drop, so your anomaly and diversity policies still have traces to work with.

Keep two rules in mind:

- **Don’t set the collector rate below your total tail target.** If the collector keeps less of the original traffic than your policies want, the tail sampler can only keep what arrives, and you fall short of your target.
- **Start high, then lower the rate gradually.** Begin with more headroom than you expect to need, then reduce the collector rate toward your cost goal while you confirm that errors and other important traces are still captured. 50% is a good starting point that balances egress and processing savings with tail sampling effectiveness. Because the tail policy is absolute, you can change the collector rate without editing your policies.

## Keep span metrics accurate

Sampling shouldn’t distort the metrics you derive from your traces. Because sampling now happens in your environment, Grafana Cloud no longer receives every trace and can’t build metrics by counting the traces it sees. Instead, it extrapolates back to the original population using the sampling probability recorded in each trace’s trace state.

Each kept trace carries an adjusted count, which is the inverse of the probability with which it was kept. If the collector kept 20% of traffic, each kept trace has an adjusted count of `1 / 0.20 = 5` and counts as 5 toward the request, error, and duration metrics. Adding up adjusted counts, rather than counting raw traces, reconstructs the metrics for 100% of the original traffic even though only a fraction of traces reach Grafana Cloud. You lose stored example traces, but not metric accuracy, so your span metric dashboards and alerts stay accurate even when you sample aggressively at the collector.

> Note
> 
> Trace state based span metric extrapolation isn’t enabled by default. To have Tempo extrapolate span metrics from the trace state for your traffic, contact Grafana Support and ask to enable trace state based span metric extrapolation for your Tempo tenant. Until it’s enabled, span metrics reflect only the sampled traffic that reaches Grafana Cloud, not the original population.

### Extrapolate in TraceQL metrics queries

The extrapolation above applies to the span metrics that Grafana Cloud generates for you. When you run [TraceQL metrics queries](/docs/tempo/latest/metrics-from-traces/metrics-queries/) yourself, they compute metrics on demand from the sampled traces stored in Tempo, so their results reflect only the sampled traffic. Add the experimental `with(extrapolate=true)` query hint to scale the result back to the original traffic. Tempo reads the sampling probability from each span’s trace state and scales that span’s contribution by `1 / sampling_probability`.

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name="api" } | rate() with(extrapolate=true)
```

The hint applies to `rate`, `count_over_time`, `sum_over_time`, `avg_over_time`, `histogram_over_time`, `quantile_over_time`, and `compare`. It doesn’t change `min_over_time` or `max_over_time`, because extremes don’t scale with sampling. For more detail, refer to the [Tempo metrics functions documentation](/docs/tempo/next/metrics-from-traces/metrics-queries/functions/).

## Summary

Multi-tier sampling reduces the egress and processing cost of your traces in your own environment while Adaptive Traces keeps making the final decision in Grafana Cloud. The OpenTelemetry trace state carries the sampling probability the collector applied, which keeps the two tiers consistent, keeps tail sampling rates absolute against your original traffic, and lets Grafana Cloud extrapolate span metrics back to 100%.