Tracing is one of the richest observability signals, but it's also noisy and susceptible to data bloat. 

In a busy system, the vast majority of traces describe the same healthy, fast, successful request over and over, so most organizations downsample their traces to cut costs. But that approach has consequences, since the sampling strategy you choose determines whether you get a faithful picture of your whole system, or just a smaller, blurrier copy of your busiest endpoints.

We want to help you get the most from your traces without having to make too many compromises. With Adaptive Traces, you get smart sampling integrated directly in Grafana Cloud Traces. And with the new volumetric policy in Adaptive Traces, you can further optimize what you keep so you can save money and still get a diverse, representative set of traces.

The volumetric policy

The volumetric policy dynamically chooses the best traces for your sampling budget. It does this intelligently and under the hood, ensuring fair representation for all of your services, and removing the toil of creating and maintaining bespoke sampling policies. All you have to do is specify a percentage target, and the rest is done for you.

Below, we’ll talk about how we arrived at this policy, and why you need to make it part of your sampling strategy.

What's wrong with doing probabilistic sampling?

Probabilistic sampling is the easiest way to sample a random subset of traces because it uses the traceID to capture a random and uniform subset of traces. It's a very simple, widely adopted technique that can be run at any point in the tracing pipeline. But because it only uses the traceID, it is indifferent to what your traces actually contain. And that indifference is a problem, since it's akin to using a sledgehammer when you really need a scalpel.

In most systems traces are very unbalanced: high-frequency services or endpoints can emit magnitudes more spans than low-frequency services or endpoints. When you sample a flat percentage of all your traces, you preserve this imbalance in the downsampled traces.

To illustrate how this could play out, let's look at a hypothetical scenario:

The checkout service is the most used service in your system. It emits around 950 traces/sec; all other services combined emit another 50 traces/sec.

Your goal is to downsample traces to end up with roughly 100 traces/s.

We can use a probabilistic sampling policy at a flat 10%. That would mean the checkout service gets to store 95 traces/sec, and all the other services only get 5 traces/sec.

With only 5 traces/sec for all non-checkout services, it will become increasingly difficult to find rare traces.

We can improve this by manually tuning our policies. Since the checkout service is responsible for the majority of traces and most of these will be the same successful requests, we can downsample it more aggressively. Let's create a targeted probabilistic sampling policy that samples 6% of traces from the checkout service only. This policy will sample 57 traces/sec and leaves more budget for the other low-frequency services. To maximize the budget, all the other services can be sampled at 86%. This can be done by creating a second policy sampling all non-checkout traces at 80%.

This example is overtly simplified, but this kind of imbalances occur along several dimensions:

• An endpoint can have lots of successful requests (status code = 200) compared to a small rate of non-successful requests.
• One region of your service might be more popular, drowning out traces from newer, smaller regions.
• A large customer can generate more traffic than all other customers together.

Most systems have a few dominant traces and a long tail of rare, low frequency traces—but that doesn't mean there isn't value in those less-common traces.

Once you identify an imbalance, you can correct it by creating more targeted policies. Policies that sample high traffic traces at a lower percentage and keep more of the low traffic traces. The issue is constantly identifying these imbalances and addressing them. This becomes a tedious task as traffic patterns change, new services are added, and old ones are deprecated.

Dynamic sampling can do this for you, automatically and constantly adjusting.

What is dynamic sampling?

Dynamic sampling uses attributes on the traces itself to categorize traces and sample them at different sample rates. POST /checkout and Run inventory lookup become separate populations with their own frequency and budget. The dynamic sampling continuously calculates individual sample rates to optimize the desired overall target volume. It's the same process as manually tuning your policies, except it is fully automated.

Let's look at another hypothetical to see how this might work:

There are two services with very different traffic patterns: the item service is called at high rates by users browsing the site. The checkout service is called less frequently.

Since the item service generates so many traces, we can downsample it quite aggressively without impacting the discoverability. Even if we only sample 10% of traces, we still have a good signal to work with. The checkout service on the other hand is called more rarely and as a result it doesn't take up a lot of budget. Maybe we don't even need to downsample it?

Dynamic sampling is a step up from probabilistic sampling: it distributes its sampling budget so high-volume traffic doesn't crowd out everything else. 

A service with 10 times the traffic of another doesn't need to get 10 times the sampled traces. We can downsample it more so quieter services remain present in the sampled data. The result is a sampled dataset in which every service has a voice. You still get plenty of your busiest paths, but not at the expense of erasing the rest. 

We've been working hard to make advanced sampling more accessible in Adaptive Traces, and the volumetric policy is the next step in this mission.

Automating dynamic sampling with volumetric

The volumetric policy, which manages dynamic sampling out of the box, is a drop-in replacement for the probabilistic policy. If you already have Adaptive Traces configured, you can upgrade with a single click. And if you are new to Adaptive Traces, the onboarding flow will set you up with a volumetric policy from the get-go!

The volumetric policy is a full end-to-end managed experience: it automatically picks attributes to categorize traces and then balances traffic according to those categories. And this is a continuous process, constantly adjusting for traffic changes.

• Picking attributes. Volumetrically analyzes traces before any downsampling and tracks the cardinality of each attribute. Attributes with a very high cardinality are not well suited for dynamic sampling since they create too many small buckets. Attributes with little cardinality don't allow dynamic sampling to slice and dice traces. The process will select a handful of attributes that together have a reasonable cardinality.

A few good examples of commonly selected attributes are service.name, status.code and  k8s.cluster.name.

• Balancing traffic. To balance traffic, the volumetric policy categorizes traces using the chosen attributes into buckets. A bucket could be service.name=checkout-service, status.code=200, k8s.cluster.name=prod.

 Each bucket has a frequency counter and a sample rate. Each interval the policy updates the buckets using the newly observed frequencies and calculates new sample rates. This allows the sampling policy to continuously adapt to changes in traffic.

The end result is a fully managed policy that can optimize sampled data.

Does the volumetric policy really improve what I sample?

To validate how this approach improves trace coverage, we compared information density (how much unique information you retain for every byte stored) from the probabilistic and the volumetric policies. We use Shannon entropy, a standard measure of information content, to compare spans sampled by the volumetric policy to a probabilistic policy with the same sampling percentage.

Across our testing, the volumetric policy delivers about 25% higher information density than the probabilistic policy. That means a quarter more information for the exact amount of data stored. To put it in practical terms: for the same cost you get a more diverse and more representative set of traces.

The Adaptive Traces sampling toolbox

The volumetric policy doesn't work alone. It's one of several policies in Adaptive Traces that combine to keep the right traces:

• Volumetric sampling gives you even, attribute-aware coverage across trace types for your chosen budget.
• Anomaly detection uses machine learning to detect irregularities in your system. Sampling outliers at a higher rate makes it easier to analyse anomalies in your system.
• Diversity sampling guarantees you keep at least one instance of every distinct trace type. Since it's a lighter process than volumetric it can go much broader and handle higher cardinality.
• Standard policies for latency, status, etc. allow you to specify exactly what traces must be sampled or dropped. For instance, if traces used for auditing should always be retained, a policy can specify this requirement.

Together these policies retain the traces you care about most. Adaptive Traces users drop write volume by 75% to 90%, on average. And remember, if a trace you really wanted got dropped by Adaptive Traces, it can be retrieved within 24 hours.

It is easy to get started today:

• If you're new to Adaptive Traces, our onboarding experience will give you a set of starter policies—this now includes a volumetric policy!
• If you're already using Adaptive Traces, you will see a recommendation to convert your probabilistic policy to a volumetric policy with a single click.

Either way: it’s very easy to improve your trace sampling to optimize costs, capture the most valuable traces, and reduce toil, allowing for the optimal distributed tracing experience.

Grafana Assistant is the easiest way to get started with metrics, logs, traces, dashboards, and more in Grafana Cloud. We have a generous forever-free tier and plans for every use case. Sign up for free now!