This is documentation for the next version of Loki. For the latest stable release, go to the latest version.
0002: Remote Rule Evaluation
Author: Danny Kopping (firstname.lastname@example.org)
Related issues/PRs: https://github.com/grafana/mimir/pull/1536
Thread from mailing list: N/A
ruler is a component that evaluates alerting and recording rules. Loki reuses Prometheus’ rule evaluation engine. The
ruler currently operates by initialising a
querier internally and evaluating all rules “locally” (i.e. it does not rely on any other components). Each rule group executes concurrently, and rules within the rule group are evaluated sequentially (this is an implementation detail from Prometheus).
Recording rules produce metric series which are sent to a Prometheus-compatible source. Alerting rules send notifications to Alertmanager when a condition is met. Both of these rule types can play a vital role in an organisation’s observability strategy, and so their reliable evaluation is essential.
Rule evaluations can contain expensive queries. The
ruler initialises a
querier, but the
querier does not have the capability to accelerate queries; the
query-frontend component is responsible for query acceleration through splitting, sharding, caching, and other techniques.
An expensive rule query can cause an entire
ruler instance to use excessive resources and even crash. This is highly problematic for the following reasons:
- slow rule evaluations can lead to subsequent rules in a group to be delayed or missed, leading to missing alerts or gaps in recording rule metrics
- excessive resource usage can impede the evaluation of rules for other tenants (noisy neighbour)
- faster, more efficient rule evaluation
- greater isolation between tenants
- more reliable service
This proposal does not aim to make this option the default mode of evaluation; it should be optional because it increases operational complexity.
Proposal 0: Do nothing
ruler implementation is sufficient for small installations running relatively simple or inexpensive queries.
- Nothing to be done
rulerwill remain unreliable and inefficient when used in large multi-tenant environments with expensive queries.
Proposal 1: Remote Execution
Taking inspiration from Grafana Mimir’s implementation, the
ruler would be configured to send its rule query to the
query-frontend component over gRPC. The
querier instances receiving queries from the
query-frontend (or optionally via the
query-scheduler) will handle the request and send the responses to the
query-frontend and be combined. The
ruler will receive and process these responses as if the query had been executed locally.
- Takes full advantage of Loki’s query acceleration techniques, leading to faster and more efficient rule evaluation
- Operationally simple as existing
queriersetup can be used
- Per-tenant isolation available in Loki’s query path (shuffle-sharding, per-tenant queues) can be used to reduce or eliminate the noisy neighbour problem
- Increased interdependence in components, increased cross-component networking
- Reusing the same
queriersetup can cause expensive queries to starve rule evaluations of query resources, and vice versa
- Additional complexity introduced if this setup needs to be duplicated for rule evaluations (recommended: see Other Notes section below)
If this feature were to be used in conjunction with rule-based sharding, this can present some further optimisation but also some additional challenges to consider.
rulershards by rule group by default, which means that rules can be unevenly balanced across
rulerinstances if some rule groups have more expensive queries than others. Another consequence of this is that rule groups execute sequentially, so expensive queries can cause subsequent rules in the group to be delayed or even missed. Rule groups are evaluated concurrently.
Rule-based sharding distributes rules evenly across all available
ruler instances, each in their own rule group. Consequentially, each rule that belongs to a
ruler instance will be evaluated concurrently (as they’re each in their own rule group). For tenants with hundreds or thousands of rules, this can result in large batches of queries being sent to the
query-frontend in quick succession, should they all use the same interval or happen to overlap.
Assuming the remote rule evaluation takes place on the same read path that is used to execute tenant queries, care must be taken by operators who run large multi-tenant setups to ensure that large volumes of queries can be received, queued, and processed in an acceptable timeframe. The
query-scheduler component is highly recommended in these situations, as it will enable the
query components to scale out to accommodate the load. Shuffle-sharding should also be implemented to ensure that tenants with particularly large workloads do not starve out the query resources of other tenants. Alerting should also be put in place to notify operators if rule evaluations are being routinely missed or a tenants’ query queues become full.
If rule evaluations and tenant queries are slowing each other down, the read path setup would need to be duplicated so that tenant queries and rule evaluations would not share the same query execution resources.
Rule-based sharding and remote evaluation can (and should) be implemented separately. Operators should first implement remote evaluation to improve
ruler reliability, and then further investigate rule-based sharding if rule evaluations are still being missed due to the sequential execution of rule groups, or advise their tenants to split these rule groups up.