Alert rules on Grafana Labs

Alert rule types

Mon, 13 Apr 2026 19:49:35 +0000

Alert rule types

Grafana supports two different alert rule types. Learn more about each of the alert rule types, how they work, and decide which one is best for your use case.

Grafana-managed alert rules

Grafana-managed alert rules are the most flexible alert rule type. They allow you to create alerts that can act on data from any of our supported data sources.

In addition to supporting multiple data sources, you can also add expressions to transform your data and set alert conditions. Using images in alert notifications is also supported. This is the only type of rule that allows alerting from multiple data sources in a single rule definition.

The following diagram shows how Grafana-managed alerting works.

Alert rules are created within Grafana based on one or more data sources.
Alert rules are evaluated by the Alert Rule Evaluation Engine from within Grafana.
Alerts are delivered using the internal Grafana Alertmanager.

Note:

You can also configure alerts to be delivered using an external Alertmanager; or use both internal and external alertmanagers. For more information, see Add an external Alertmanager.

Data source-managed alert rules

To create data source-managed alert rules, you must have a compatible Prometheus or Loki data source.

You can check if your data source supports rule creation via Grafana by testing the data source and observing if the Ruler API is supported.

For more information on the Ruler API, refer to Ruler API.

The following diagram shows how data source-managed alerting works.

Alert rules are created and stored within the data source itself.
Alert rules can only be created based on Prometheus data.
Alert rule evaluation and delivery is distributed across multiple nodes for high availability and fault tolerance.

Choose an alert rule type

When choosing which alert rule type to use, consider the following comparison between Grafana-managed alert rules and Grafana Mimir or Loki alert rules.

Feature	Grafana-managed alert rule	Loki/Mimir-managed alert rule
Create alert rules based on data from any of our supported data sources	Yes	No: You can only create alert rules that are based on Prometheus data. The data source must have the Ruler API enabled.
Mix and match data sources	Yes	No
Includes support for recording rules	No	Yes
Add expressions to transform your data and set alert conditions	Yes	No
Use images in alert notifications	Yes	No
Scaling	More resource intensive, depend on the database, and are likely to suffer from transient errors. They only scale vertically.	Store alert rules within the data source itself and allow for “infinite” scaling. Generate and send alert notifications from the location of your data.
Alert rule evaluation and delivery	Alert rule evaluation and delivery is done from within Grafana, using an external Alertmanager; or both.	Alert rule evaluation and alert delivery is distributed, meaning there is no single point of failure.

Note:

If you are using non-Prometheus data, we recommend choosing Grafana-managed alert rules. Otherwise, choose Grafana Mimir or Grafana Loki alert rules where possible.

Recording rules

Mon, 13 Apr 2026 19:49:35 +0000

Recording rules

Recording rules are only available for compatible Prometheus or Loki data sources.

A recording rule allows you to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series. This is useful if you want to run alerts on aggregated data or if you have dashboards that query computationally expensive expressions repeatedly.

Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.

Grafana Enterprise offers an alternative to recorded rules in the form of recorded queries that can be executed against any data source.

For more information on recording rules in Prometheus, refer to recording rules.

Queries and conditions

Mon, 13 Apr 2026 19:49:35 +0000

Queries and conditions

In Grafana, queries play a vital role in fetching and transforming data from supported data sources, which include databases like MySQL and PostgreSQL, time series databases like Prometheus, InfluxDB and Graphite, and services like Elasticsearch, AWS CloudWatch, Azure Monitor and Google Cloud Monitoring.

For more information on supported data sources, see Data sources.

The process of executing a query involves defining the data source, specifying the desired data to retrieve, and applying relevant filters or transformations. Query languages or syntaxes specific to the chosen data source are utilized for constructing these queries.

In Alerting, you define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

An alert rule consists of one or more queries and expressions that select the data you want to measure.

For more information on queries and expressions, see Query and transform data.

Data source queries

Queries in Grafana can be applied in various ways, depending on the data source and query language being used. Each data source’s query editor provides a customized user interface that helps you write queries that take advantage of its unique capabilities.

Because of the differences between query languages, each data source query editor looks and functions differently. Depending on your data source, the query editor might provide auto-completion features, metric names, variable suggestions, or a visual query-building interface.

Some common types of query components include:

Metrics or data fields: Specify the specific metrics or data fields you want to retrieve, such as CPU usage, network traffic, or sensor readings.

Time range: Define the time range for which you want to fetch data, such as the last hour, a specific day, or a custom time range.

Filters: Apply filters to narrow down the data based on specific criteria, such as filtering data by a specific tag, host, or application.

Aggregations: Perform aggregations on the data to calculate metrics like averages, sums, or counts over a given time period.

Grouping: Group the data by specific dimensions or tags to create aggregated views or breakdowns.

Note:

Grafana does not support alert queries with template variables. More information is available here.

Expression queries

In Grafana, an expression is used to perform calculations, transformations, or aggregations on the data source queried data. It allows you to create custom metrics or modify existing metrics based on mathematical operations, functions, or logical expressions.

By leveraging expression queries, users can perform tasks such as calculating the percentage change between two values, applying functions like logarithmic or trigonometric functions, aggregating data over specific time ranges or dimensions, and implementing conditional logic to handle different scenarios.

In Alerting, you can only use expressions for Grafana-managed alert rules. For each expression, you can choose from the math, reduce, and resample expressions. These are called multi-dimensional rules, because they generate a separate alert for each series.

You can also use classic condition, which creates an alert rule that triggers a single alert when its condition is met. As a result, Grafana sends only a single alert even when alert conditions are met for multiple series.

Note:

Classic conditions exist mainly for compatibility reasons and should be avoided if possible.

Reduce

Aggregates time series values in the selected time range into a single value.

Math

Performs free-form math functions/operations on time series and number data. Can be used to preprocess time series data or to define an alert condition for number data.

Resample

Realigns a time range to a new set of timestamps, this is useful when comparing time series data from different data sources where the timestamps would otherwise not align.

Threshold

Checks if any time series data matches the threshold condition.

The threshold expression allows you to compare two single values. It returns 0 when the condition is false and 1 if the condition is true. The following threshold functions are available:

Is above (x > y)
Is below (x < y)
Is within range (x > y1 AND x < y2)
Is outside range (x < y1 AND x > y2)

Classic condition

Checks if any time series data matches the alert condition.

Note:

Classic condition expression queries always produce one alert instance only, no matter how many time series meet the condition. Classic conditions exist mainly for compatibility reasons and should be avoided if possible.

Aggregations

Grafana Alerting provides the following aggregation functions to enable you to further refine your query.

These functions are available for Reduce and Classic condition expressions only.

Function	Expression	What it does
avg	Reduce / Classic	Displays the average of the values
min	Reduce / Classic	Displays the lowest value
max	Reduce / Classic	Displays the highest value
sum	Reduce / Classic	Displays the sum of all values
count	Reduce / Classic	Counts the number of values in the result
last	Reduce / Classic	Displays the last value
median	Reduce / Classic	Displays the median value
diff	Classic	Displays the difference between the newest and oldest value
diff_abs	Classic	Displays the absolute value of diff
percent_diff	Classic	Displays the percentage value of the difference between newest and oldest value
percent_diff_abs	Classic	Displays the absolute value of percent_diff
count_non_null	Classic	Displays a count of values in the result set that aren’t `null`

Alert condition

An alert condition is the query or expression that determines whether the alert will fire or not depending on the value it yields. There can be only one condition which will determine the triggering of the alert.

After you have defined your queries and/or expressions, choose one of them as the alert rule condition.

When the queried data satisfies the defined condition, Grafana triggers the associated alert, which can be configured to send notifications through various channels like email, Slack, or PagerDuty. The notifications inform you about the condition being met, allowing you to take appropriate actions or investigate the underlying issue.

By default, the last expression added is used as the alert condition.

Recovery threshold

Note
The recovery threshold feature is currently only available in OSS.

To reduce the noise of flapping alerts, you can set a recovery threshold different to the alert threshold.

Flapping alerts occur when a metric hovers around the alert threshold condition and may lead to frequent state changes, resulting in too many notifications being generated.

Grafana-managed alert rules are evaluated for a specific interval of time. During each evaluation, the result of the query is checked against the threshold set in the alert rule. If the value of a metric is above the threshold, an alert rule fires and a notification is sent. When the value goes below the threshold and there is an active alert for this metric, the alert is resolved, and another notification is sent.

It can be tricky to create an alert rule for a noisy metric. That is, when the value of a metric continually goes above and below a threshold. This is called flapping and results in a series of firing - resolved - firing notifications and a noisy alert state history.

For example, if you have an alert for latency with a threshold of 1000ms and the number fluctuates around 1000 (say 980 ->1010 -> 990 -> 1020, and so on) then each of those will trigger a notification.

To solve this problem, you can set a (custom) recovery threshold, which basically means having two thresholds instead of one. An alert is triggered when the first threshold is crossed and is resolved only when the second threshold is crossed.

For example, you could set a threshold of 1000ms and a recovery threshold of 900ms. This way, an alert rule will only stop firing when it goes under 900ms and flapping is reduced.

Alert instances

Mon, 13 Apr 2026 19:49:35 +0000

Alert instances

Grafana managed alerts support multi-dimensional alerting. Each alert rule can create multiple alert instances. This is exceptionally powerful if you are observing multiple series in a single expression.

Consider the following PromQL expression:

sum by(cpu) (
  rate(node_cpu_seconds_total{mode!="idle"}[1m])
)

A rule using this expression will create as many alert instances as the amount of CPUs we are observing after the first evaluation, allowing a single rule to report the status of each CPU.

Namespaces, folders, and groups

Mon, 13 Apr 2026 19:49:35 +0000

Namespaces, folders, and groups

Alerts can be organized using folders for Grafana-managed rules and namespaces for Mimir or Loki rules and group names.

Namespaces and folders

When creating Grafana-managed rules, the folder can be used to perform access control and grant or deny access to all rules within a specific folder.

A namespace contains one or more groups. The rules within a group are run sequentially at a regular interval. The default interval is one (1) minute. You can rename Grafana Mimir or Loki rule namespaces and groups, and edit group evaluation intervals.

Groups

The rules within a group are run sequentially at a regular interval, meaning no rules will be evaluated at the same time and in order of appearance. The default interval is one (1) minute. You can rename Grafana Mimir or Loki rule namespaces and groups, and edit group evaluation intervals.

Note If you want rules to be evaluated concurrently and with different intervals, consider storing them in different groups.

Note Grafana managed alert rules are evaluated concurrently instead of sequentially.

Alert rule evaluation

Mon, 13 Apr 2026 19:49:35 +0000

Alert rule evaluation

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.

Evaluation group

Every alert rule is part of an evaluation group. Each evaluation group contains an evaluation interval that determines how frequently the alert rule is checked.

Data-source managed alert rules within the same group are evaluated one after the other, while alert rules in different groups can be evaluated simultaneously. This feature is especially useful when you want to ensure that recording rules are evaluated before any alert rules.

Grafana-managed alert rules are evaluated at the same time, regardless of alert rule group. The default evaluation interval is set at 10 seconds, which means that Grafana-managed alert rules are evaluated every 10 seconds to the closest 10-second window on the clock, for example, 10:00:00, 10:00:10, 10:00:20, and so on. You can also configure your own evaluation interval, if required.

Note:

Evaluation groups and alerts grouping in notification policies are two separate things. Grouping in notification policies allows multiple alerts sharing the same labels to be sent in the same time message.

Pending period

By setting a pending period, you can avoid unnecessary alerts for temporary problems.

In the pending period, you select the period in which an alert rule can be in breach of the condition until it fires.

Example

Imagine you have an alert rule evaluation interval set at every 30 seconds and the pending period to 90 seconds.

Evaluation will occur as follows:

[00:30] First evaluation - condition not met.

[01:00] Second evaluation - condition breached. Pending counter starts. Alert starts pending.

[01:30] Third evaluation - condition breached. Pending counter = 30s. Pending state.

[02:00] Fourth evaluation - condition breached. Pending counter = 60s Pending state.

[02:30] Fifth evaluation - condition breached. Pending counter = 90s. Alert starts firing

If the alert rule has a condition that needs to be in breach for a certain amount of time before it takes action, then its state changes as follows:

When the condition is first breached, the rule goes into a “pending” state.
The rule stays in the “pending” state until the condition has been broken for the required amount of time - pending period.
Once the required time has passed, the rule goes into a “firing” state.
If the condition is no longer broken during the pending period, the rule goes back to its normal state.

Note:

If you want to skip the pending state, you can simply set the pending period to 0. This effectively skips the pending period and your alert rule will start firing as soon as the condition is breached.

When an alert rule fires, alert instances are produced, which are then sent to the Alertmanager.

State and health of alert rules

Mon, 13 Apr 2026 19:49:35 +0000

State and health of alert rules

The state and health of alert rules help you understand several key status indicators about your alerts.

There are three key components: alert rule state, alert instance state, and alert rule health. Although related, each component conveys subtly different information.

Alert rule state

An alert rule can be in either of the following states:

State	Description
Normal	None of the time series returned by the evaluation engine is in a `Pending` or `Firing` state.
Pending	At least one time series returned by the evaluation engine is `Pending`.
Firing	At least one time series returned by the evaluation engine is `Firing`.

Note
Alerts will transition first to pending and then firing, thus it will take at least two evaluation cycles before an alert is fired.

Alert instance state

An alert instance can be in either of the following states:

State	Description
Normal	The state of an alert that is neither firing nor pending, everything is working correctly.
Pending	The state of an alert that has been active for less than the configured threshold duration.
Alerting	The state of an alert that has been active for longer than the configured threshold duration.
NoData	No data has been received for the configured time window.
Error	The error that occurred when attempting to evaluate an alert rule.

Alert rule health

An alert rule can have one the following health statuses:

State	Description
Ok	No error when evaluating an alert rule.
Error	An error occurred when evaluating an alert rule.
NoData	The absence of data in at least one time series returned during a rule evaluation.

Special alerts for `NoData` and `Error`

When evaluation of an alert rule produces state NoData or Error, Grafana Alerting will generate alert instances that have the following additional labels:

Label	Description
alertname	Either `DatasourceNoData` or `DatasourceError` depending on the state.
datasource_uid	The UID of the data source that caused the state.

You can handle these alerts the same way as regular alerts by adding a silence, route to a contact point, and so on.

Notification templating

Mon, 13 Apr 2026 19:49:35 +0000

Notification templating

Notifications sent via contact points are built using notification templates. Grafana’s default templates are based on the Go templating system where some fields are evaluated as text, while others are evaluated as HTML (which can affect escaping).

The default template default_template.go is a useful reference for custom templates.

Since most of the contact point fields can be templated, you can create reusable custom templates and use them in multiple contact points.

Using templates

The following example shows how to use default templates to render an alert message in Slack. The message title contains a count of alerts that are firing or were resolved. The message body lists the alerts and their status.

The following example shows the use of a custom template within one of the contact point fields.

Nested templates

You can embed templates within other templates.

For example, you can define a template fragment using the define keyword:

{{ define "mytemplate" }}
  {{ len .Alerts.Firing }} firing. {{ len .Alerts.Resolved }} resolved.
{{ end }}

You can then embed custom templates within this fragment using the template keyword. For example:

Alert summary:
{{ template "mytemplate" . }}

You can use any of the following built-in template options to embed custom templates.

Name	Notes
`default.title`	Displays high-level status information.
`default.message`	Provides a formatted summary of firing and resolved alerts.
`teams.default.message`	Similar to `default.message`, formatted for Microsoft Teams.

HTML in notification templates

HTML in alerting notification templates is escaped. We do not support rendering of HTML in the resulting notification.

Some notifiers support alternative methods of changing the look and feel of the resulting notification. For example, Grafana installs the base template for alerting emails to <grafana-install-dir>/public/emails/ng_alert_notification.html. You can edit this file to change the appearance of all alerting emails.

Alert rules on Grafana Labs

Alert rule types

Alert rule types

Grafana-managed alert rules

Data source-managed alert rules

Choose an alert rule type

Recording rules

Recording rules

Queries and conditions

Queries and conditions

Data source queries

Expression queries

Aggregations

Alert condition

Recovery threshold

Alert instances

Alert instances

Namespaces, folders, and groups

Namespaces, folders, and groups

Namespaces and folders

Groups

Alert rule evaluation

Alert rule evaluation

Evaluation group

Pending period

State and health of alert rules

State and health of alert rules

Alert rule state

Alert instance state

Alert rule health

Special alerts for NoData and Error

Notification templating

Notification templating

Using templates

Nested templates

HTML in notification templates

Special alerts for `NoData` and `Error`