Best practices for policies

Adaptive Traces gives you the power to control your observability costs and reduce noise by keeping only the traces that provide value. An effective policy configuration is key to achieving this balance.

This guide outlines the best practices for creating and evaluating policies.

The core evaluation logic

Before building, it’s crucial to understand how policies are evaluated.

Drop always wins: If a trace matches any policy configured to drop it, it is immediately discarded. This rule has the highest priority and overrides all others.
Keep if any match: If a trace is not dropped, it only needs to match one keep policy to be saved. Think of your keep policies as a large “OR” condition.
Default is drop: If a trace matches no policies at all (neither keep nor drop), it is discarded.

Best Practice: Build your policies strategically

Don’t create rules randomly. Follow a layered approach for the best results.

Set a probabilistic keep policy

Set a probabilistic (or percentage-based) sampling rule to give you a representative sample of your “normal” day-to-day traffic without incurring the cost of keeping everything.

Start Low: Begin with a low percentage, such as 5% or 10%. You can always increase it later if you find you need more data.
Example: Create a Probabilistic policy to keep 10% of all remaining traces.

Define specific keep policies

Ensure you never lose your most critical signals. These policies guarantee that traces related to incidents, errors, and poor performance are always captured.

Always keep errors: The most important rule. Keep 100% of traces that have a status code of Error.
Capture high latency: Define what slow means for your key services and create policies to keep any trace that exceeds that threshold, for example, latency > 2 seconds.
Example:

Create a Status Code policy to keep all traces where the status is ERROR.

Create drop policies

Eliminate high-volume, low-value traffic.

Example: Create a String Attribute policy to drop all traces where http.target is /grafana.

Best Practice: Continuously review and refine

As your application and traffic patterns changes, your policies should, too.

Monitor policy impact: Use metrics to see how many traces each policy is keeping or dropping. If a rule is unexpectedly noisy, tighten its conditions.
Use precise attributes: Avoid overly broad rules. Instead of matching a URL with a containsoperator, use equals on a specific attribute like http.target or rpc.method for more predictable results.