Documentation Grafana Cloud Adaptive Telemetry Adaptive Traces Guides Policy evaluation example

How policies are evaluated

When a trace is received, it is evaluated against all your policies at once to make a single, final decision: keep (sample) or drop. The logic follows two core principles.

Principle 1: Keep if it matches any keep policy (the “or” logic)

Think of your keep policies as a giant OR condition. A trace only needs to match one of these policies to be flagged for keeping. It does not need to match all of them.

For example, if you have these policies:

• Keep traces with errors.
• Keep traces with latency > 2 seconds.
• Keep 10% of all remaining traces.

A trace with an error is kept.

A trace with a 3-second latency is kept. It does not matter if the slow trace also had an error; one match is enough.

The 10% probabilistic policy acts as a catch-all for traces that do not match any of your more specific rules.

Principle 2: Drop always wins

A drop policy is an absolute veto. If a trace matches any drop policy, it is discarded immediately, even if it also matches one or more keep policies.

For example, imagine you add this policy:

• Drop all traces from the /grafana endpoint.

If a slow, erroneous trace comes from the /grafana endpoint, it is dropped. The drop rule takes priority over all keep rules.

Principle 3: Policies are evaluated using sample on first match

Adaptive Traces evaluates sampling policies using a sample-on-first-match model.

This means that as soon as a policy decides to either sample or drop a trace, evaluation stops and no further policies are considered.

As a result, a later policy may appear to sample nothing even though it is configured correctly. This happens when an earlier, more general policy has already sampled the trace. Because the trace was already decided earlier in the chain, the later policy never gets a chance to evaluate it, so its sampled metric remains at zero.

This can look like the policy is not working, but in reality the trace matched a different policy earlier and was handled there.

Example

• A top-level default probabilistic policy samples 10% of all traces.

• A more specific Auth API policy also samples 10% of traces.

• If the default policy samples the trace first, the Auth API policy is skipped entirely.

Policy order is not guaranteed

When evaluating traces, Adaptive Traces does not guarantee a specific ordering of keep policies. The effective evaluation order is essentially random and may not match either the configuration order or the order shown in the UI.

The only exception is drop policies, which are always evaluated first because they take priority and act as an immediate veto.

Example

Let’s look at how the system evaluates a few different traces with the policies from above.

In summary, just remember these rules.

• A trace is evaluated against all policies at once. A trace is kept if it matches at least one keep policy.
• A trace is dropped if it matches any drop policy.
• If a trace matches both a keep and a drop policy, drop always wins.

Policy example using code

The following code snippet is example of how the policies above can be defined using JSON.

Copy

# Policy 1: Keep traces with latency > 2 seconds

{
  "threshold_ms": 2000,
}


# Policy 2: Keep 10% of all traces as a general sample

{
  "sampling_percentage": 10
}

# Policy 3: Drop all traces from the /grafana endpoint

{
    "drop_sub_policy": [
    {
      "name": "healthcheck-drop",
      "type": "string_attribute",
      "string_attribute": {
        "key": "http.target",
        "values": [
          "/grafana"
        ]
      }
    }
  ]
}

# Policy 4: Keep all traces that have an error

{
      "name": "error-keep",
      "type": "status_code",
 "status_codes": [
    "ERROR"
  ]
}

How to interpret this code

1. List of Policies: The system evaluates a trace against this entire list.
2. Drop: The healthcheck-drop policy is explicitly configured to action: drop. When a trace with http.target: /grafana is processed, this rule provides the definitive drop decision that overrides any keep decisions from other policies.
3. Keep: For a trace that is not dropped, the system only needs one of the other policies. For example, high-latency-keep, error-keep, or probabilistic-catch-all to return a keep decision.