DocumentationGrafana documentationAlertingConfigure alert rulesConfigure recording rules

Configure recording rules

Recording rules calculate frequently needed expressions or computationally expensive expressions in advance and save the result as a new set of time series. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.

For more information on recording rules in Prometheus, refer to Defining recording rules in Prometheus.

Recording rules are run as instant rules, which means that they run every 10s. To overwrite this configuration, update the min_interval in your custom configuration file.

min_interval sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules will be adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.

This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.

Configure data source-managed recording rules

Configure data source-managed recording rules.

Before you begin

• Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.

• For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.

  • Loki - The local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.

  • Grafana Mimir - use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.

To configure data-source managed recording rules, complete the following steps.

1. Click Alerts & IRM -> Alerting -> Alert rules.
2. Scroll to the Data source-managed section and click +New recording rule.

Enter recording rule name

The recording rule name must be a Prometheus metric name and contain no whitespace.

Define recording rule

Select your data source and enter a query.

Add namespace and group

1. From the Namespace dropdown, select an existing rule namespace or add a new one.

   Namespaces can contain one or more rule groups and only have an organizational purpose.

2. From the Group dropdown, select an existing group within the selected namespace or add a new one.

   Newly created rules are appended to the end of the group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.

Add labels

1. Add custom labels selecting existing key-value pairs from the drop down, or add new labels by entering the new key or value.

2. Click Save rule to save the rule or Save rule and exit to save the rule and go back to the Alerting page.

Configure Grafana-managed recording rules

Configure Grafana-managed recording rules.

Before you begin

• Enable the grafanaManagedRecordingRules feature flag.

To configure Grafana-managed recording rules, complete the following steps.

1. Click Alerts & IRM -> Alerting -> Alert rules.
2. Scroll to the Grafana-managed section and click +New recording rule.

Enter a recording rule and metric name

Enter a names to identify your recording rule and metric. The metric name must be a Prometheus metric name and contain no whitespace.

For more information, refer to Metrics and labels.

Define recording rule

Define a query to get the data you want to measure and a condition that needs to be met before an alert rule fires.

1. Select a data source.
2. From the Options dropdown, specify a time range.

1. Add a query.

   To add multiple queries, click Add query.

   All alert rules are managed by Grafana by default. If you want to switch to a data source-managed alert rule, click Switch to data source-managed alert rule.

2. Add one or more [expressions].

   a. For each expression, select either Classic condition to create a single alert rule, or choose from the Math, Reduce, and Resample options to generate separate alert for each series.

   Note

   When using Prometheus, you can use an instant vector and built-in functions, so you don’t need to add additional expressions.

   b. Click Preview to verify that the expression is successful.

3. To add a recovery threshold, turn the Custom recovery threshold toggle on and fill in a value for when your alert rule should stop firing.

   You can only add one recovery threshold in a query and it must be the alert condition.

4. Click Set as alert condition on the query or expression you want to set as your alert condition.

Set evaluation behavior

Use alert rule evaluation to determine how frequently an alert rule should be evaluated and how quickly it should change its state.

To do this, you need to make sure that your alert rule is in the right evaluation group and set a pending period time that works best for your use case.

1. Select a folder or click + New folder.

2. Select an evaluation group or click + New evaluation group.

   If you are creating a new evaluation group, specify the interval for the group.

   All rules within the same group are evaluated concurrently over the same time interval.

3. Enter a pending period.

   The pending period is the period in which an alert rule can be in breach of the condition until it fires.

   Once a condition is met, the alert goes into the Pending state. If the condition remains active for the duration specified, the alert transitions to the Firing state, else it reverts to the Normal state.

4. Turn on pause alert notifications, if required.

   Note

   You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts. Pausing stops alert rule evaluation and doesn’t create any alert instances. This is different to mute timings, which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.

Add labels

Add labels to your rule for searching, silencing, or routing to a notification policy.