Documentation Grafana Cloud Adaptive Telemetry Adaptive Metrics Create rules Understand aggregation delay

Understand the aggregation delay

To generate aggregated samples, Adaptive Metrics buffers incoming samples for a period of time defined by the aggregation delay. The aggregation delay defaults to 90 seconds and is overridden on a per-rule basis via the aggregation_delay field. This allows time for datapoints to arrive from different sources with differing latency or other factors that delay ingestion. During the aggregation delay, which defaults to 90 seconds, samples are not available for querying. This can have some impact on query results.

The most typical impact of aggregation is a small gap on the right side of any graph displaying aggregated data.

Depending on the query powering your panel, you might see other artifacts while querying recent data. For example:

• A drop in rate or other “over_time” queries.
• A spike in queries that mix aggregated and unaggregated data.

In most cases, these effects are minor and only rarely appear.

Understand the impact on recording and alerting rules

While the effects of aggregation delay are usually small, there is one case where it has a large impact. When evaluating a recording rule, any gap or small deviation due to aggregation delay persists permanently. When evaluating an alerting rule, those discrepancies can trigger alerting thresholds and lead to false positive or false negative alerts.

Here’s a panel with the actual query next to the rule. Notice how the rule shows a much lower value than the query.

When a recording or alerting rule is evaluated, the most recent aggregated samples are not yet available for querying. As a result, the recording or alerting rule executes without access to the complete dataset, which can lead to lower quality results. Additionally, aggregation delay is unpredictable; although it typically remains around 90 seconds with default settings, it can occasionally spike unexpectedly.

For this reason, it’s not recommended to run recording or alerting rules on data processed by Adaptive Metrics. The recommendations engine does not recommend an aggregation rule that could affect your existing recording or alerting rules. Additionally, it suggests removing any manually created aggregation rules that conflict with your current recording or alerting rules.