Create Grafana-managed recording rules

Thu, 05 Dec 2024 18:25:49 +0000

Create Grafana-managed recording rules

Recording rules allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.

Grafana does not contain an embedded time-series database to store recording rule results. You must bring your own Prometheus-compatible database to store the series generated by recording rules.

Grafana-managed recording rules offer the same Prometheus-like semantics but allow you to query data sources supported by alerting. Additionally, you can use recording rules to import and map data from other data sources into Prometheus.

Note
Grafana-managed recording rules are enabled by default in Grafana Cloud.

In Grafana OSS and Enterprise, you must enable them by following the Before you begin instructions.

To configure Grafana-managed recording rules, complete the following steps.

Before you begin

This section only applies to Grafana OSS and Grafana Enterprise.

First, enable the grafanaManagedRecordingRules feature flag.

Then, enable the feature by setting enabled = true in the [recording_rules] section of the Grafana config .ini. Provide the URL of your Prometheus-compatible remote-write endpoint in the url field, along with optional credentials or headers.

[recording_rules]
enabled = true
url = http://my-example-prometheus.local:9090/api/prom/push
basic_auth_username = my-user
basic_auth_password = my-pass

[recording_rules.custom_headers]
X-My-Header = MyValue

Add new recording rule

To create a new Grafana-managed recording rule:

Click Alerts & IRM -> Alerting -> Alert rules.
Scroll to the Grafana-managed section and click +New recording rule.
Enter the names to identify your recording rule and metric.

The metric name must be a Prometheus metric name and contain no whitespace. For details, refer to Prometheus metric names.

Define recording rule

Define a query to get the data you want to measure and set the recording rule output.

Select a data source.
From the Options dropdown, specify a time range.

Note
Grafana Alerting only supports fixed relative time ranges, for example, now-24hr: now.

It does not support absolute time ranges: 2021-12-02 00:00:00 to 2021-12-05 23:59:592 or semi-relative time ranges: now/d to: now.
Add a query.

To add multiple queries, click Add query.
Add one or more expressions.

a. For each expression, select either Classic condition to create a single recording rule, or choose from the Math, Reduce, and Resample options.

When using Prometheus, you can use an instant vector and built-in functions, so you don’t need to add additional expressions.

b. Click Preview to verify that the expression is successful.
Click Set as recording rule output on the query or expression you want to set as your rule output.

Set evaluation behavior

Use recording rule evaluation to determine how frequently a recording rule should be evaluated.

To do this, you need to make sure that your recording rule is in the right evaluation group with an interval that works best for your use case.

Select a folder or click + New folder.
Select an evaluation group or click + New evaluation group.

If you are creating a new evaluation group, specify the interval for the group.

All rules within the same group are evaluated concurrently over the same time interval. Every recording rule in a group uses the same evaluation time, meaning that all queries from the same group are always aligned with each other.
Before or after creating a recording rule, you have the option to Pause evaluation if necessary.

Advanced configuration

min_interval sets the minimum interval to enforce between rule evaluations. The default value is 10s which equals the scheduler interval. Rules are adjusted if they are less than this value or if they are not multiple of the scheduler interval (10s). Higher values can help with resource management as fewer evaluations are scheduled over time.

This setting has precedence over each individual rule frequency. If a rule frequency is lower than this value, then this value is enforced.

This setting applies to both Grafana-managed alert and recording rules.

Add labels

Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.

Query the new metric in dashboards or alert rules

Click Save rule or Save rule and exit to save the rule.

Once saved, the new recording metric is available for use in dashboards and alert rules.

Create data source-managed recording rules

Mon, 16 Dec 2024 11:24:34 +0000

Create data source-managed recording rules

Recording rules allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric.

Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.

Data source-managed recording rules can query Prometheus-based data sources like Mimir or Loki. For more information on recording rules in Prometheus, refer to Defining recording rules in Prometheus.

Note that in data source-managed groups, the alert rules and recording rules within the same evaluation group are evaluated sequentially. This is useful to ensure that a recording rule is evaluated before any other alert rule queries the pre-computed metric.

Before you begin

Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update Grafana Mimir managed alerting rules.
For Grafana Mimir and Loki data sources, enable the ruler API by configuring their respective services.
- Loki - The local rule storage type, default for the Loki data source, supports only viewing of rules. To edit rules, configure one of the other rule storage types.
- Mimir - use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.

Add new recording rule

To create a new data source-managed recording rule:

Click Alerts & IRM -> Alerting -> Alert rules.
Scroll to the Data source-managed section and click +New recording rule.

Enter recording rule name

The recording rule name must be a Prometheus metric name and contain no whitespace.

Define recording rule

Select your data source and enter a query. The queries used in data source-managed recording rules always run as instant queries.

Add namespace and group

From the Namespace dropdown, select an existing rule namespace or add a new one.

Namespaces can contain one or more rule groups and only have an organizational purpose.
From the Group dropdown, select an existing group within the selected namespace or add a new one.

Rules within a group are run sequentially at a regular interval, with the same evaluation time.

Newly created rules are appended to the end of the group, and you can reorder them from the Alert rules page.

Add labels

Optionally, you can add custom labels to the resulting metric by selecting existing key-value pairs from the drop down or entering the new key or value.

Query the new metric in dashboards or alert rules

Click Save rule or Save rule and exit to save the rule.

Once saved, the new recording metric is available for use in dashboards and alert rules.

Create recording rules on Grafana Labs

Create Grafana-managed recording rules

Create Grafana-managed recording rules

Before you begin

Add new recording rule

Define recording rule

Set evaluation behavior

Advanced configuration

Add labels

Query the new metric in dashboards or alert rules

Create data source-managed recording rules

Create data source-managed recording rules

Before you begin

Add new recording rule

Enter recording rule name

Define recording rule

Add namespace and group

Add labels

Query the new metric in dashboards or alert rules