Troubleshoot discarded raw samples

Adaptive Metrics enforces constraints on the timestamps of incoming raw samples that are destined for aggregation. Specifically, all samples must arrive in Grafana Cloud no later than the aggregation delay to be processed successfully. Each aggregated series tracks the aggregation delay separately according to the latest sample. This means that an overall lag in sample ingestion is tolerable, as long as it affects all ingested samples equally. However, a large clock skew between incoming samples can cause some samples that are either too new or too old to be rejected.

Understand the types of discarded samples

In your Grafana Cloud stack, in the GrafanaCloud folder, go to the dashboard called Usage Insights - 5 - Metrics Ingestion.

The Discarded Samples / second panel shows the reasons that samples are discarded. For example:

aggregator-sample-too-old - this refers to samples that arrive too far into the future. Specifically, if the sample timestamp is more than one minute into the future, according to the current wall clock time in distributors.
sample-too-new-for-aggregation - this refers to samples that arrive too late to be aggregated. Specifically, this happens if the sample timestamp is older than the aggregation delay of the most recently received sample for the corresponding aggregated series.

Debug the `aggregator-sample-too-old` error

Typically, the aggregator-sample-too-old error is caused by one or more hosts that are lagging behind in ingesting samples in Grafana Cloud. In your Grafana Cloud stack, in the GrafanaCloud folder, go to the dashboard called Usage Insights - 5 - Metrics Ingestion.

The Ingestion Error Details panel shows a sample of recent errors, including the raw series label and timestamp that you can use to identify the source of the lagging data.

Debug the `sample-too-new-for-aggregation` error

Typically, the sample-too-new-for-aggregation error is caused by one or more hosts with system clocks in the future. Grafana Labs recommends configuring NTP to keep each host’s clock synchronized.

In your Grafana Cloud stack, in the GrafanaCloud folder, go to the dashboard titled Usage Insights - 5 - Metrics Ingestion.

The Ingestion Error Details panel shows a sample of recent errors, including the raw series label and timestamp that you can use to identify the source of the future data.