Documentation Grafana Cloud Observe and act Optimize data Adaptive Traces Guides Policy evaluation example

How policies are evaluated

When a trace is received, it is evaluated against your policies to make a single, final decision: keep (sample) or drop.

You rarely need to reason about this logic if you use the out-of-the-box recommendations; they’re designed to work together. Understanding the evaluation model matters most when you add your own custom policies. Refer to Best practices for policies before you do.

Principle 1: Keep if it matches any keep policy (the “or” logic)

Think of your keep policies as a giant OR condition. A trace only needs to match one of these policies to be flagged for keeping. It does not need to match all of them.

For example, if you have these policies:

• Keep traces with errors.
• Keep traces with latency > 2 seconds.
• Keep 10% of all remaining traces.

A trace with an error is kept.

A trace with a 3-second latency is kept. It does not matter if the slow trace also had an error; one match is enough.

The 10% probabilistic policy acts as a catch-all for traces that do not match any of your more specific rules.

Principle 2: Drop always wins

A drop policy is an absolute veto. If a trace matches any drop policy, it is discarded immediately, even if it also matches one or more keep policies.

For example, imagine you add this policy:

• Drop all traces from the /grafana endpoint.

If a slow, erroneous trace comes from the /grafana endpoint, it is dropped. The drop rule takes priority over all keep rules.

Principle 3: Policies are evaluated using sample on first match

Adaptive Traces evaluates sampling policies using a sample-on-first-match model.

This means that as soon as a policy decides to either sample or drop a trace, evaluation stops and no further policies are considered.

As a result, a later policy may appear to sample nothing even though it is configured correctly. This happens when an earlier, more general policy has already sampled the trace. Because the trace was already decided earlier in the chain, the later policy never gets a chance to evaluate it, so its sampled metric remains at zero.

This can look like the policy is not working, but in reality the trace matched a different policy earlier and was handled there.

Example

• A top-level default probabilistic policy samples 10% of all traces.

• A more specific Auth API policy also samples 10% of traces.

• If the default policy samples the trace first, the Auth API policy is skipped entirely.

Policy order

Adaptive Traces evaluates policy types in a fixed priority order:

1. Drop policies, which take priority and act as an immediate veto
2. Anomaly policies
3. Diversity policies
4. Volumetric policies
5. All other policies

Within that last group, the order of your custom keep policies is not guaranteed and may not match the order shown in the UI. Don’t build configurations that depend on one custom policy being evaluated before another.

Because of this ordering and the sample-on-first-match model, the recommended volumetric, diversity, and anomaly policies make their decisions before your custom keep policies are considered. A custom policy’s sampled count only reflects traces that earlier policies didn’t already decide on.

Example

Let’s look at how the system evaluates a few different traces with the policies from above.

In summary, just remember these rules.

• A trace is kept if it matches at least one keep policy.
• A trace is dropped if it matches any drop policy.
• If a trace matches both a keep and a drop policy, drop always wins.
• Evaluation stops at the first policy that samples or drops a trace, and drop, anomaly, diversity, and volumetric policies are always evaluated before your custom policies.

Policy example using code

The following code snippet is example of how the policies above can be defined using JSON.

Copy

# Policy 1: Keep traces with latency > 2 seconds

{
  "threshold_ms": 2000
}


# Policy 2: Keep 10% of all traces as a general sample

{
  "sampling_percentage": 10
}

# Policy 3: Drop all traces from the /grafana endpoint

{
    "drop_sub_policy": [
    {
      "name": "healthcheck-drop",
      "type": "string_attribute",
      "string_attribute": {
        "key": "http.target",
        "values": [
          "/grafana"
        ]
      }
    }
  ]
}

# Policy 4: Keep all traces that have an error

{
      "name": "error-keep",
      "type": "status_code",
 "status_codes": [
    "ERROR"
  ]
}

How to interpret this code

1. List of Policies: The system evaluates a trace against this entire list.
2. Drop: The healthcheck-drop policy is explicitly configured to action: drop. When a trace with http.target: /grafana is processed, this rule provides the definitive drop decision that overrides any keep decisions from other policies.
3. Keep: For a trace that is not dropped, the system only needs one of the other policies. For example, high-latency-keep, error-keep, or probabilistic-catch-all to return a keep decision.