How policies are evaluated
When a trace is received, it is evaluated against all your policies at once to make a single, final decision: keep (sample) or drop. The logic follows two core principles.
Principle 1: Keep if it matches any keep policy (the “or” logic)
Think of your keep policies as a giant OR condition. A trace only needs to match one of these policies to be flagged for keeping. It does not need to match all of them.
For example, if you have these policies:
- Keep traces with errors.
- Keep traces with latency > 2 seconds.
- Keep 10% of all remaining traces.
A trace with an error is kept.
A trace with a 3-second latency is kept. It does not matter if the slow trace also had an error; one match is enough.
The 10% probabilistic policy acts as a catch-all for traces that do not match any of your more specific rules.
Principle 2: Drop always wins
A drop policy is an absolute veto. If a trace matches any drop policy, it is discarded immediately, even if it also matches one or more keep policies.
For example, imagine you add this policy:
- Drop all traces from the
/grafana
endpoint.
If a slow, erroneous trace comes from the /grafana endpoint
, it is dropped. The drop rule takes priority over all keep rules.
Example
Let’s look at how the system evaluates a few different traces with the policies from above.
Trace Details | Evaluation | Final Decision |
---|---|---|
Trace A: duration: 3s , endpoint: /api/checkout | Matches the latency > 2s keep policy. | KEEP |
Trace B: duration: 150ms , endpoint: /api/products | Doesn’t match any specific keep or drop policy. It’s now subject to the 10% probabilistic rule. | KEEP (if it’s in the 10%) or DROP (if it’s in the other 90%) |
Trace C: duration: 3s , endpoint: /grafana | Matches both the latency > 2s (keep) policy AND the /grafana (drop) policy. | DROP (The drop policy wins) |
In summary, just remember these rules.
- A trace is evaluated against all policies at once. A trace is kept if it matches at least one keep policy.
- A trace is dropped if it matches any drop policy.
- If a trace matches both a keep and a drop policy, drop always wins.
Policy example using code
The following code snippet is example of how the policies above can be defined using JSON.
# Policy 1: Keep traces with latency > 2 seconds
{
"threshold_ms": 2000,
}
# Policy 2: Keep 10% of all traces as a general sample
{
"sampling_percentage": 10
}
# Policy 3: Drop all traces from the /grafana endpoint
{
"drop_sub_policy": [
{
"name": "healthcheck-drop",
"type": "string_attribute",
"string_attribute": {
"key": "http.target",
"values": [
"/grafana"
]
}
}
]
}
# Policy 4: Keep all traces that have an error
{
"name": "error-keep",
"type": "status_code",
"status_codes": [
"ERROR"
]
}
How to interpret this code
- List of Policies: The system evaluates a trace against this entire list.
- Drop: The
healthcheck-drop
policy is explicitly configured to action: drop. When a trace withhttp.target: /grafana
is processed, this rule provides the definitive drop decision that overrides any keep decisions from other policies. - Keep: For a trace that is not dropped, the system only needs one of the other policies. For example,
high-latency-keep
,error-keep
, orprobabilistic-catch-all
to return a keep decision.