Documentation Grafana documentation Alerting Examples Dynamic thresholds

Example of dynamic thresholds per dimension

In Grafana Alerting, each alert rule supports only one condition expression.

That’s enough in many cases—most alerts use a fixed numeric threshold like latency > 3s or error_rate > 5% to determine their state.

As your alerting setup grows, you may find that different targets require different threshold values.

Instead of duplicating alert rules, you can assign a different threshold value to each target—while keeping the same condition. This simplifies alert maintenance.

This example shows how to do that using multi-dimensional alerts and a Math expression.

Example overview

You’re monitoring latency across multiple API services. Initially, you want to get alerted if the 95th percentile latency (p95_api_latency) exceeds 3 seconds, so your alert rule uses a single static threshold:

Copy

p95_api_latency > 3

But the team quickly finds that some services require stricter thresholds. For example, latency for payment APIs should stay under 1.5s, while background jobs can tolerate up to 5s. The team establishes different thresholds per service:

• p95_api_latency{service="checkout-api"}: must stay under 1.5s.
• p95_api_latency{service="auth-api"}: also strict, 1.5s.
• p95_api_latency{service="catalog-api"}: less critical, 3s.
• p95_api_latency{service="async-tasks"}: background jobs can tolerate up to 5s.

You want to avoid creating one alert rule per service—this is harder to maintain.

In Grafana Alerting, you can define one alert rule that monitors multiple similar components like this scenario. This is called multi-dimensional alerts: one alert rule, many alert instances—one per unique label set.

But there’s an issue: Grafana supports only one alert condition per rule.

Copy

One alert rule
├─ One condition ( e.g., $A > 3)
│  └─ Applies to all returned series in $A
│     ├─ {service="checkout-api"}
│     ├─ {service="auth-api"}
│     ├─ {service="catalog-api"}
│     └─ {service="async-tasks"}

To evaluate per-service thresholds, you need a distinct threshold value for each returned series.

Dynamic thresholds using a Math expression

You can create a dynamic alert condition by operating on two queries with a Math expression.

• $A for query results (e.g., p95_api_latency).
• $B for per-service thresholds (from CSV data or another query).
• $A > $B is the Math expression that defines the alert condition.

Grafana evaluates the Math expression per series, by joining series from $A and $B based on their shared labels before applying the expression.

Here’s an example of an arithmetic operation:

In practice, you must align your threshold input with the label sets returned by your alert query.

The following table illustrates how a per-service threshold is evaluated in the previous example:

In this example:

• $A comes from the p95_api_latency query.
• $B is manually defined with a threshold value for each series in $A.
• The alert condition compares $A>$B using a Math relational operator (e.g., >, <, >=, <=, ==, !=) that joins series by matching labels.
• Grafana evaluates the alert condition and sets the firing state where the condition is true.

The Math expression works as long as each series in $A can be matched with exactly one series in $B. They must align in a way that produces a one-to-one match between series in $A and $B.

Labels in both series don’t need to be identical. If labels are a subset of the other, they can join. For example:

• $A returns series {host="web01", job="event"} 30 and {host="web02", job="event"} 20.
• $B returns series {host="web01"} 10 and {host="web02"} 0.
• $A + $B returns {host="web01", job="event"} 40 and {host="web02", job="event"} 20.

Try it with TestData

You can use the TestData data source to replicate this example:

1. Add the TestData data source through the Connections menu.

2. Create an alert rule.

   Navigate to Alerting → Alert rules and click New alert rule.

3. Simulate a query ($A) that returns latencies for each service.

   Select TestData as the data source and configure the scenario.

   • Scenario: Random Walk

   • Alias: latency

   • Labels: service=api-$seriesIndex

   • Series count: 4

   • Start value: 1

   • Min: 1, Max: 4

     This uses $seriesIndex to assign unique service labels: api-0, api-1, etc.

4. Define per-service thresholds with static data.

   Add a new query ($B) and select TestData as the data source.

   From Scenario, select CSV Content and paste this CSV:

   Copy

    service,value
    api-0,1.5
    api-1,1.5
    api-2,3
    api-3,5

   The service column must match the labels from $A.

   The value column is a numeric value used for the alert comparison.

   For details on CSV format requirements, see table data examples.

5. Add a new Reduce expression ($C).

   • Type: Reduce
   • Input: A
   • Function: Mean
   • Name: C

   This calculates the average latency for each service: api-0, api-1, etc.

6. Add a new Math expression.

   • Type: Math
   • Expression: $C > $B
   • Set this expression as the alert condition.

   This fires if the average latency ($C) exceeds the threshold from $B for any service.

7. Preview the alert.

   Tip

   You can explore this alerting example in Grafana Play.

   Open the example to view alert evaluation results, generated alert instances, and alert rule details.

Dynamic thresholds using a SQL expression

Instead of a Math expression, you can define dynamic thresholds with a SQL expression.

With SQL expressions, you don’t need a second query for the thresholds. The SQL expression can compare each series against its own threshold and returns the alert condition result.

The following table illustrates the evaluation:

You can define the thresholds inside the query with a CASE statement:

SQL Copy

SELECT
  A.service AS service,
  CASE
    WHEN A.__value__ > CASE A.service
                     WHEN 'checkout-api' THEN 1.5
                     WHEN 'auth-api' THEN 1.5
                     WHEN 'catalog-api' THEN 3
                     WHEN 'sync-work' THEN 5
                     ELSE 2.0 -- default threshold
                   END
    THEN 1 ELSE 0
  END AS alert_result
FROM A

In this example:

• A is the data returned by the p95 latency query.

• The inner CASE assigns a threshold to each service. Unlike the Math expression, the ELSE clause lets unlisted services use the default threshold of 2.0 instead of being dropped from the result.

• The outer CASE compares the latency A.__value__ against the threshold and returns returns 1 when the latency exceeds the threshold, and 0 otherwise.

• The service column becomes the label of each alert instance and the alert_result column is the numeric output used as the alert condition.

  The output of the SQL expression must follow the table data format: only one numeric column, which Grafana uses to evaluate the alert condition. You can return multiple non-numeric columns, each of which becomes a label of the alert instances.

  The SQL expression is the alert condition, so it sets the firing state when alert_result is 1.

Try it with TestData

You can use the TestData data source to replicate this example:

1. Add the TestData data source through the Connections menu.

2. Create an alert rule.

   Navigate to Alerting → Alert rules and click New alert rule.

3. Simulate a query ($A) that returns latencies for each service.

   Select TestData as the data source and configure the scenario. From Scenario, select CSV Content and paste this CSV:

   Copy

    service, __value__
    checkout-api, 5
    auth-api,  1
    catalog-api, 3
    sync-work, 8
    public-api, 3

4. Add a SQL expression ($B) to define the dynamic threshold.

   Click Add expression, select SQL, and paste this query:

   SQL Copy

    SELECT
      A.service AS service,
      CASE
        WHEN A.__value__ > CASE A.service
                        WHEN 'checkout-api' THEN 1.5
                        WHEN 'auth-api' THEN 1.5
                        WHEN 'catalog-api' THEN 3
                        WHEN 'sync-work' THEN 5
                        ELSE 2.0 -- default threshold
                      END
        THEN 1 ELSE 0
      END AS alert_result
    FROM A

5. Set the SQL expression ($B) as the alert condition.

6. Preview the alert.

   Tip

   You can explore this alerting example in Grafana Play.

   Open the example to view alert evaluation results, generated alert instances, and alert rule details.

Join thresholds from another query

Instead of defining static thresholds in the SQL expression, you can retrieve them from another query:

B Copy

service, threshold
checkout-api, 1.5
auth-api, 2
catalog-api, 4
sync-work, 5

To combine both queries, use a LEFT JOIN between them on a shared column such as service:

SQL Copy

SELECT
  A.service AS service,
  CASE
    WHEN A.__value__ > COALESCE(B.threshold, 2) THEN 1
    ELSE 0
  END AS alert_result
FROM A
LEFT JOIN B ON A.service = B.service

In this example:

• A is the data returned by the p95 latency query.
• B is the data returned by the threshold query. It must include the service column and a threshold column.
• LEFT JOIN keeps every row from A and matches it with the row in B that has the same service.
• COALESCE(B.threshold, 2) uses 2 as the default threshold when B has no row for a service. Without a LEFT JOIN and a default, services without a threshold are dropped from the result.
• The service column becomes the label of each alert instance, and alert_result is the numeric output used as the alert condition.

Combine multiple conditions

This example showed how to build a single alert rule with different thresholds per series using multi-dimensional alerts.

This approach scales well when monitoring similar components with distinct reliability goals. By aligning series from two queries, you can apply a dynamic threshold—one value per label set—without duplicating rules.

The same technique works in other scenarios to build more advanced threshold logic by combining multiple conditions—such as latency, error rate, or traffic volume. For example, you can define a PromQL expression that sets a latency threshold which adjusts based on traffic—allowing higher response times during periods of high-load.

Copy

(
  // Fires when p95 latency > 2s during usual traffic (≤ 1000 req/s)
  service:latency:p95 > 2 and service:request_rate:rate1m <= 1000
)
or
(
  // Fires when p95 latency > 4s during high traffic (> 1000 req/s)
  service:latency:p95 > 4 and service:request_rate:rate1m > 1000
)