Understand the aggregation delay

To generate aggregated samples, Adaptive Metrics buffers incoming samples for a period of time defined by the aggregation delay. The aggregation delay defaults to 90 seconds and is overridden on a per-rule basis via the aggregation_delay field. This allows time for datapoints to arrive from different sources with differing latency or other factors that delay ingestion. During the aggregation delay, which defaults to 90 seconds, samples are not available for querying. This can have some impact on query results.

The most typical impact of aggregation is a small gap on the right side of any graph displaying aggregated data.

Depending on the query powering your panel, you might see other artifacts while querying recent data. For example:

A drop in rate or other “over_time” queries.
A spike in queries that mix aggregated and unaggregated data.

In most cases, these effects are minor and only rarely appear.

Understand the impact on recording and alerting rules

While the effects of aggregation delay are usually small, there is one case where it has a large impact. When evaluating a recording rule, any gap or small deviation due to aggregation delay persists permanently. When evaluating an alerting rule, those discrepancies can trigger alerting thresholds and lead to false positive or false negative alerts.

Here’s a panel with the actual query next to the rule. Notice how the rule shows a much lower value than the query.

At the time the rule is evaluated, the most recent aggregated samples are unavailable for querying. The rule query runs without the full dataset present, and therefore, produces a low-quality result.

For this reason, it’s not recommended to run recording or alerting rules on aggregated data. The recommendations engine never recommends a rule that would affect your existing rules, and it does recommend removing any manually created rules that conflict with existing rules.