---
title: "Example of dynamic thresholds per dimension | Grafana documentation"
description: "This example shows how to use a distinct threshold value per dimension using multi-dimensional alerts and a Math expression."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Example of dynamic thresholds per dimension

In Grafana Alerting, each alert rule supports only one condition expression.

That’s enough in many cases—most alerts use a fixed numeric threshold like `latency > 3s` or `error_rate > 5%` to determine their state.

As your alerting setup grows, you may find that different targets require different threshold values.

Instead of duplicating alert rules, you can assign a **different threshold value to each target**—while keeping the same condition. This simplifies alert maintenance.

This example shows how to do that using [multi-dimensional alerts](https://grafana.com/docs/grafana/v13.3/alerting/examples/multi-dimensional-alerts/) and a [Math expression](https://grafana.com/docs/grafana/v13.3/alerting/fundamentals/alert-rules/queries-conditions/#math).

## Example overview

You’re monitoring latency across multiple API services. Initially, you want to get alerted if the 95th percentile latency (`p95_api_latency`) exceeds 3 seconds, so your alert rule uses a single static threshold:

![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```none
p95_api_latency > 3
```

But the team quickly finds that some services require stricter thresholds. For example, latency for payment APIs should stay under 1.5s, while background jobs can tolerate up to 5s. The team establishes different thresholds per service:

- `p95_api_latency{service="checkout-api"}`: must stay under `1.5s`.
- `p95_api_latency{service="auth-api"}`: also strict, `1.5s`.
- `p95_api_latency{service="catalog-api"}`: less critical, `3s`.
- `p95_api_latency{service="async-tasks"}`: background jobs can tolerate up to `5s`.

You want to avoid creating one alert rule per service—this is harder to maintain.

In Grafana Alerting, you can define one alert rule that monitors multiple similar components like this scenario. This is called [multi-dimensional alerts](https://grafana.com/docs/grafana/v13.3/alerting/examples/multi-dimensional-alerts/): one alert rule, many alert instances—**one per unique label set**.

But there’s an issue: Grafana supports only **one alert condition per rule**.

![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```none
One alert rule
├─ One condition ( e.g., $A > 3)
│  └─ Applies to all returned series in $A
│     ├─ {service="checkout-api"}
│     ├─ {service="auth-api"}
│     ├─ {service="catalog-api"}
│     └─ {service="async-tasks"}
```

To evaluate per-service thresholds, you need a distinct threshold value for each returned series.

## Dynamic thresholds using a Math expression

You can create a dynamic alert condition by operating on two queries with a [Math expression](https://grafana.com/docs/grafana/v13.3/alerting/fundamentals/alert-rules/queries-conditions/#math).

- `$A` for query results (e.g., `p95_api_latency`).
- `$B` for per-service thresholds (from CSV data or another query).
- `$A > $B` is the *Math* expression that defines the alert condition.

Grafana evaluates the *Math* expression **per series**, by joining series from `$A` and `$B` based on their shared labels before applying the expression.

Here’s an example of an arithmetic operation:

- `$A` returns series `{host="web01"} 30` and `{host="web02"} 20`.
- `$B` returns series `{host="web01"} 10` and `{host="web02"} 0`.
- `$A + $B` returns `{host="web01"} 40` and `{host="web02"} 20`.

In practice, you must align your threshold input with the label sets returned by your alert query.

The following table illustrates how a per-service threshold is evaluated in the previous example:

Expand table

| $A: p95 latency query        | $B: threshold value            | $C: $A>$B                 | State      |
|------------------------------|--------------------------------|------------------------------|------------|
| `{service="checkout-api"} 3` | `{service="checkout-api"} 1.5` | `{service="checkout-api"} 1` | **Firing** |
| `{service="auth-api"} 1`     | `{service="auth-api"} 1.5`     | `{service="auth-api"} 0`     | **Normal** |
| `{service="catalog-api"} 2`  | `{service="catalog-api"} 3`    | `{service="catalog-api"} 0`  | **Normal** |
| `{service="sync-work"} 3`    | `{service="sync-work"} 5`      | `{service="sync-work"} 0`    | **Normal** |

In this example:

- `$A` comes from the `p95_api_latency` query.
- `$B` is manually defined with a threshold value for each series in `$A`.
- The alert condition compares `$A>$B` using a *Math* relational operator (e.g., `>`, `<`, `>=`, `<=`, `==`, `!=`) that joins series by matching labels.
- Grafana evaluates the alert condition and sets the firing state where the condition is true.

The *Math* expression works as long as each series in `$A` can be matched with exactly one series in `$B`. They must align in a way that produces a one-to-one match between series in `$A` and `$B`.

> Caution
> 
> If a series in one query doesn’t match any series in the other, it’s excluded from the result and a warning message is displayed:
> 
> *1 items **dropped from union(s)**: \["$A &gt; $B": ($B: {service=payment-api})]*

**Labels in both series don’t need to be identical**. If labels are a subset of the other, they can join. For example:

- `$A` returns series `{host="web01", job="event"}` 30 and `{host="web02", job="event"}` 20.
- `$B` returns series `{host="web01"}` 10 and `{host="web02"}` 0.
- `$A` + `$B` returns `{host="web01", job="event"}` 40 and `{host="web02", job="event"}` 20.

### Try it with TestData

You can use the [TestData data source](https://grafana.com/docs/grafana/v13.3/datasources/testdata/) to replicate this example:

1. Add the **TestData** data source through the **Connections** menu.
2. Create an alert rule.
   
   Navigate to **Alerting** → **Alert rules** and click **New alert rule**.
3. Simulate a query (`$A`) that returns latencies for each service.
   
   Select **TestData** as the data source and configure the scenario.
   
   - Scenario: Random Walk
   - Alias: latency
   - Labels: service=api-$seriesIndex
   - Series count: 4
   - Start value: 1
   - Min: 1, Max: 4
     
     This uses `$seriesIndex` to assign unique service labels: `api-0`, `api-1`, etc.
   
   [](https://grafana.com/media/docs/alerting/example-dynamic-thresholds-latency-series-v2.png)
4. Define per-service thresholds with static data.
   
   Add a new query (`$B`) and select **TestData** as the data source.
   
   From **Scenario**, select **CSV Content** and paste this CSV:
   
   ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```none
    service,value
    api-0,1.5
    api-1,1.5
    api-2,3
    api-3,5
   ```
   
   The `service` column must match the labels from `$A`.
   
   The `value` column is a numeric value used for the alert comparison.
   
   For details on CSV format requirements, see [table data examples](https://grafana.com/docs/grafana/v13.3/alerting/examples/table-data/).
5. Add a new **Reduce** expression (`$C`).
   
   - Type: Reduce
   - Input: A
   - Function: Mean
   - Name: C
   
   This calculates the average latency for each service: `api-0`, `api-1`, etc.
6. Add a new **Math** expression.
   
   - Type: Math
   - Expression: `$C > $B`
   - Set this expression as the **alert condition**.
   
   This fires if the average latency (`$C`) exceeds the threshold from `$B` for any service.
7. **Preview** the alert.
   
   [Alert preview evaluating multiple series with distinct threshold values using a Math expression](https://grafana.com/media/docs/alerting/example-dynamic-thresholds-preview-v3.png)
   
   > Tip
   > 
   > You can explore this [**alerting example in Grafana Play**](https://play.grafana.org/alerting/grafana/dynamic-thresholds/view?tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=alert-dynamic-thresholds).
   > 
   > Open the example to view alert evaluation results, generated alert instances, and alert rule details.

## Dynamic thresholds using a SQL expression

Instead of a Math expression, you can define dynamic thresholds with a [SQL expression](https://grafana.com/docs/grafana/v13.3/visualizations/panels-visualizations/query-transform-data/sql-expressions/).

With SQL expressions, you don’t need a second query for the thresholds. The SQL expression can compare each series against its own threshold and returns the alert condition result.

The following table illustrates the evaluation:

Expand table

| $A: p95 latency query        | $B: Threshold | $B: SQL result               | State      |
|------------------------------|---------------|------------------------------|------------|
| `{service="checkout-api"} 3` | `1.5`         | `{service="checkout-api"} 1` | **Firing** |
| `{service="auth-api"} 1`     | `1.5`         | `{service="auth-api"} 0`     | **Normal** |
| `{service="catalog-api"} 2`  | `3`           | `{service="catalog-api"} 0`  | **Normal** |
| `{service="sync-work"} 3`    | `5`           | `{service="sync-work"} 0`    | **Normal** |

You can define the thresholds inside the query with a `CASE` statement:

SQL ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```sql
SELECT
  A.service AS service,
  CASE
    WHEN A.__value__ > CASE A.service
                     WHEN 'checkout-api' THEN 1.5
                     WHEN 'auth-api' THEN 1.5
                     WHEN 'catalog-api' THEN 3
                     WHEN 'sync-work' THEN 5
                     ELSE 2.0 -- default threshold
                   END
    THEN 1 ELSE 0
  END AS alert_result
FROM A
```

In this example:

- `A` is the data returned by the p95 latency query.
- The inner `CASE` assigns a threshold to each `service`. Unlike the Math expression, the `ELSE` clause lets unlisted services use the default threshold of `2.0` instead of being dropped from the result.
- The outer `CASE` compares the latency `A.__value__` against the threshold and returns returns `1` when the latency exceeds the threshold, and `0` otherwise.
- The `service` column becomes the label of each alert instance and the `alert_result` column is the numeric output used as the **alert condition**.
  
  > The output of the SQL expression must follow the [table data format](https://grafana.com/docs/grafana/v13.3/alerting/examples/table-data/): only one numeric column, which Grafana uses to evaluate the alert condition. You can return multiple non-numeric columns, each of which becomes a label of the alert instances.
  
  The SQL expression is the **alert condition**, so it sets the firing state when `alert_result` is `1`.

### Try it with TestData

You can use the [TestData data source](https://grafana.com/docs/grafana/v13.3/datasources/testdata/) to replicate this example:

1. Add the **TestData** data source through the **Connections** menu.
2. Create an alert rule.
   
   Navigate to **Alerting** → **Alert rules** and click **New alert rule**.
3. Simulate a query (`$A`) that returns latencies for each service.
   
   Select **TestData** as the data source and configure the scenario. From **Scenario**, select **CSV Content** and paste this CSV:
   
   ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```none
    service, __value__
    checkout-api, 5
    auth-api,  1
    catalog-api, 3
    sync-work, 8
    public-api, 3
   ```
4. Add a SQL expression (`$B`) to define the dynamic threshold.
   
   Click **Add expression**, select **SQL**, and paste this query:
   
   SQL ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```sql
    SELECT
      A.service AS service,
      CASE
        WHEN A.__value__ > CASE A.service
                        WHEN 'checkout-api' THEN 1.5
                        WHEN 'auth-api' THEN 1.5
                        WHEN 'catalog-api' THEN 3
                        WHEN 'sync-work' THEN 5
                        ELSE 2.0 -- default threshold
                      END
        THEN 1 ELSE 0
      END AS alert_result
    FROM A
   ```
5. Set the SQL expression (`$B`) as the **alert condition**.
6. **Preview** the alert.
   
   [Alert preview evaluating multiple series with distinct threshold values using an SQL expression](https://grafana.com/media/docs/alerting/example-dynamic-thresholds-with-sql-expression-preview.png)
   
   > Tip
   > 
   > You can explore this [**alerting example in Grafana Play**](https://play.grafana.org/alerting/grafana/dynamic-thresholds-sql-expression/view?tech=docs&pg=alerting-examples&plcmt=callout-tip&cta=alert-dynamic-thresholds-sql-expression).
   > 
   > Open the example to view alert evaluation results, generated alert instances, and alert rule details.

### Join thresholds from another query

Instead of defining static thresholds in the SQL expression, you can retrieve them from another query:

B ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```b
service, threshold
checkout-api, 1.5
auth-api, 2
catalog-api, 4
sync-work, 5
```

To combine both queries, use a `LEFT JOIN` between them on a shared column such as `service`:

SQL ![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```sql
SELECT
  A.service AS service,
  CASE
    WHEN A.__value__ > COALESCE(B.threshold, 2) THEN 1
    ELSE 0
  END AS alert_result
FROM A
LEFT JOIN B ON A.service = B.service
```

In this example:

- `A` is the data returned by the p95 latency query.
- `B` is the data returned by the threshold query. It must include the `service` column and a `threshold` column.
- `LEFT JOIN` keeps every row from `A` and matches it with the row in `B` that has the same `service`.
- `COALESCE(B.threshold, 2)` uses `2` as the default threshold when `B` has no row for a service. Without a `LEFT JOIN` and a default, services without a threshold are dropped from the result.
- The `service` column becomes the label of each alert instance, and `alert_result` is the numeric output used as the **alert condition**.

## Combine multiple conditions

This example showed how to build a single alert rule with different thresholds per series using [multi-dimensional alerts](https://grafana.com/docs/grafana/v13.3/alerting/examples/multi-dimensional-alerts/).

This approach scales well when monitoring similar components with distinct reliability goals. By aligning series from two queries, you can apply a dynamic threshold—one value per label set—without duplicating rules.

The same technique works in other scenarios to build more advanced threshold logic by combining multiple conditions—such as latency, error rate, or traffic volume. For example, you can define a PromQL expression that sets a latency threshold which adjusts based on traffic—allowing higher response times during periods of high-load.

![Copy code to clipboard](https://grafana.com/media/images/icons/icon-copy-small-2.svg) Copy

```none
(
  // Fires when p95 latency > 2s during usual traffic (≤ 1000 req/s)
  service:latency:p95 > 2 and service:request_rate:rate1m <= 1000
)
or
(
  // Fires when p95 latency > 4s during high traffic (> 1000 req/s)
  service:latency:p95 > 4 and service:request_rate:rate1m > 1000
)
```