---
title: "Outlier detection examples | Grafana Cloud documentation"
description: "This guide demonstrates common outlier detection patterns for comparing similar resources and identifying those that behave differently from their peers."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Outlier detection examples

This guide demonstrates common [outlier detection](/docs/grafana-cloud/ai-tools/dynamic-alerting/outlier-detection/) patterns for comparing similar resources and identifying those that behave differently from their peers.

1. Detect uneven traffic distribution across replicas.
2. Detect unusual resource utilization across pods.
3. Use different outlier detection algorithms depending on how the group behaves.

## Detect HTTP traffic imbalance

This example detects when one pod handles a disproportionate share of HTTP traffic compared with its peers.

The outlier detection query calculates each pod’s percentage of the total service traffic:

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
# Returns one series per pod representing its share of the total service traffic
100 * sum by (pod) (
  rate(http_requests_total{service_name="$service-name"}[$__rate_interval])
)
/ scalar(sum(
  rate(http_requests_total{service_name="$service-name"}[$__rate_interval])
))
```

The outlier detector uses the [**MAD** algorithm](/docs/grafana-cloud/ai-tools/dynamic-alerting/outlier-detection/#choose-the-detection-algorithm) because pods are expected to maintain a stable signal over time. The load-balancing configuration aims to distribute requests evenly across the replicas, so a pod that continuously receives significantly more or less traffic than its peers is an indication of an imbalance.

**MAD** compares each group member against a stable baseline derived from the group’s recent behavior, making it well suited to identifying a pod whose traffic share differs from its peers.

If traffic increases or decreases across all replicas at the same time, their relative shares can remain similar and no outlier is detected. **A pod is identified as an outlier when its traffic share differs significantly from the rest of the group**.

[](/media/docs/grafana-cloud/machine-learning/screenshot-traffic-imbalance-outlier-detection-example.png)

> Tip
> 
> You can explore this [outlier detector example in Grafana Play](https://play.grafana.org/a/grafana-ml-app/outlier-detector/5ff46e7e-5ef1-45e2-8203-10927b44bc0c?tech=docs&pg=ml-examples&plcmt=callout-tip&cta=traffic-imbalance-outlier-detector-example).

## Detect resource utilization outliers

You can use outlier detection to compare resource utilization across similar Kubernetes pods and identify a pod whose behavior differs from its peers.

In this demo, memory and CPU utilization behave differently, so these examples use different [outlier detection algorithms](/docs/grafana-cloud/ai-tools/dynamic-alerting/outlier-detection/#choose-the-detection-algorithm):

- **MAD** for memory utilization because it’s expected to stay close to a stable baseline, so comparing each pod against that baseline is a reliable way to spot one that drifts away from it.
- **DBSCAN** for CPU utilization because it moves with workload, so comparing pods with each other at each point in time adapts as the whole group shifts together, rather than comparing against a fixed baseline.

### Detect memory utilization outliers

This example compares memory utilization across pods, expressed as a percentage of each pod’s configured memory limit.

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
# Returns one series per pod with memory utilization as a percentage of its limit
100 *
max by (pod) (
  container_memory_working_set_bytes{...}
)
/
max by (pod) (
  kube_pod_container_resource_limits{resource="memory", ...}
)
```

The outlier detector uses the [**MAD** algorithm](/docs/grafana-cloud/ai-tools/dynamic-alerting/outlier-detection/#choose-the-detection-algorithm) because memory utilization is expected to remain stable over days.

An outlier does not necessarily indicate a resource problem. Differences can result from uneven workload, cache state, application behavior, or pod lifecycle. Use the outlier signal as additional context when investigating unusual memory behavior.

[](/media/docs/grafana-cloud/machine-learning/screenshot-memory-outlier-detection-example.png)

> Tip
> 
> You can explore this [outlier detector example in Grafana Play](https://play.grafana.org/a/grafana-ml-app/outlier-detector/1805ff3b-65e5-47df-9b3c-d8f44521b6a9?tech=docs&pg=ml-examples&plcmt=callout-tip&cta=memory-outlier-detector-example).

### Detect CPU utilization outliers

This example compares CPU utilization across pods, expressed as a percentage of requested CPU.

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
# Returns one series per pod with CPU utilization as a percentage of requested CPU
100 * sum by (pod) (
  rate(container_cpu_usage_seconds_total{...}[$__rate_interval])
)
/
sum by (pod) (
  kube_pod_container_resource_requests{resource="cpu", ...}
)
```

The outlier detector uses the [**DBSCAN** algorithm](/docs/grafana-cloud/ai-tools/dynamic-alerting/outlier-detection/#choose-the-detection-algorithm) because CPU utilization often changes with application workload. DBSCAN compares pods against each other at each point in time, so if several pods consume more CPU at once during a traffic increase, the group’s accepted range shifts with them instead of flagging every pod as an outlier.

[](/media/docs/grafana-cloud/machine-learning/screenshot-cpu-outlier-detection-example.png)

> Tip
> 
> You can explore this [outlier detector example in Grafana Play](https://play.grafana.org/a/grafana-ml-app/outlier-detector/e474dc93-d83b-4bd0-9416-d791d50ea0f5?tech=docs&pg=ml-examples&plcmt=callout-tip&cta=cpu-outlier-detector-example).

## Alert on outliers

An outlier indicates that a resource is behaving differently from its peers, but not every outlier requires an alert. Use alerts when a situation is [actionable or requires investigation](/docs/grafana-cloud/observe-and-act/alert-and-measure-reliability/alerting/guides/best-practices/).

In this example, the memory utilization detector provides the outlier signal for each pod. The alert evaluates the `:outliers` metric and fires when a pod remains classified as an outlier for a sustained period.

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
max by (pod) (
  memory_utilization:outliers
)
```

[](/media/docs/grafana-cloud/machine-learning/screenshot-memory-outlier-detection-alert-example.png)

> Tip
> 
> You can explore [this alert example in Grafana Play](https://play.grafana.org/alerting/grafana/cfu360yzc8qgwd/view?tech=docs&pg=ml-examples&plcmt=callout-tip&cta=memory-outlier-detection-alert-example).

The alert is intentionally an investigative signal. An outlier does not necessarily indicate memory pressure or an imminent OOM kill. Review the affected pod’s memory trend and actual utilization to determine whether further investigation is needed.

For more information about querying outlier metrics and creating alerts, including alerting on a percentage of the group, refer to [Query and alert on outliers](/docs/grafana-cloud/ai-tools/dynamic-alerting/outlier-detection/query-and-alerting/#alerting-on-outliers).