Loki Improvement Documents (LIDs) on Grafana Labs

0001: Introducing LIDs

Thu, 10 Apr 2025 12:15:54 +0000

0001: Introducing LIDs

Author: Danny Kopping (danny.kopping@grafana.com)

Date: 01/2023

Sponsor(s): @dannykopping

Type: Process

Status: Accepted

Related issues/PRs: N/A

Thread from mailing list: N/A

Background

As the Grafana Loki project grows, we have seen more and more contributions from external (outside Grafana Labs) contributors.

Problem Statement

Many of these external contributions are large and complex, and have taken these contributors significant time to implement. Large contributions that are made without prior discussion with maintainers are at risk of being rejected if they are misguided, implemented inefficiently, or simply undesired; this is obviously suboptimal both for the contributors and the maintainers.

Aside from external contributions, changes being proposed by Grafana Loki maintainers may also require community engagement before being worked on.

Goals

It would be preferable to engage with contributors before they make large contributions to ensure that both their and the project’s interests are aligned. The community at large must also have a voice when feature or process changes are being proposed, to protect their own interests.

We should implement a lightweight process that guides the implementation of major changes to the project.

Proposals

Proposal 0: Do nothing

We will continue to attract large, often complex, external contributions that have not be discussed with maintainers prior to the work being put in; this may lead to suboptimal outcomes for the relationship between the project and its community.

Proposal 1: Loki Improvement Documents

Inspired by Python’s PEP and Kafka’s KIP approaches, we should create a process for formally documenting improvements to Loki which are permanently viewable, and document our decisions.

Other Notes

Google Docs were considered for this, but they are less useful because:

they would need to be owned by the Grafana Labs organisation, so that they remain viewable even if the author closes their account
we already have previous design documents in our documentation and, in a recent (5th Jan 2023) community call, the community expressed a preference for this type of approach

0002: Remote Rule Evaluation

Wed, 06 Sep 2023 12:47:18 +0000

0002: Remote Rule Evaluation

Author: Danny Kopping (danny.kopping@grafana.com)

Date: 01/2023

Sponsor(s): @dannykopping

Type: Feature

Status: Accepted

Related issues/PRs: https://github.com/grafana/mimir/pull/1536

Thread from mailing list: N/A

Background

The ruler is a component that evaluates alerting and recording rules. Loki reuses Prometheus’ rule evaluation engine. The ruler currently operates by initialising a querier internally and evaluating all rules “locally” (i.e. it does not rely on any other components). Each rule group executes concurrently, and rules within the rule group are evaluated sequentially (this is an implementation detail from Prometheus).

Recording rules produce metric series which are sent to a Prometheus-compatible source. Alerting rules send notifications to Alertmanager when a condition is met. Both of these rule types can play a vital role in an organisation’s observability strategy, and so their reliable evaluation is essential.

Problem Statement

Rule evaluations can contain expensive queries. The ruler initialises a querier, but the querier does not have the capability to accelerate queries; the query-frontend component is responsible for query acceleration through splitting, sharding, caching, and other techniques.

An expensive rule query can cause an entire ruler instance to use excessive resources and even crash. This is highly problematic for the following reasons:

slow rule evaluations can lead to subsequent rules in a group to be delayed or missed, leading to missing alerts or gaps in recording rule metrics
excessive resource usage can impede the evaluation of rules for other tenants (noisy neighbour)

Goals

faster, more efficient rule evaluation
greater isolation between tenants
more reliable service

Non-Goals

This proposal does not aim to make this option the default mode of evaluation; it should be optional because it increases operational complexity.

Proposals

Proposal 0: Do nothing

Loki’s current ruler implementation is sufficient for small installations running relatively simple or inexpensive queries.

Pros:

Nothing to be done

Cons:

Loki’s ruler will remain unreliable and inefficient when used in large multi-tenant environments with expensive queries.

Proposal 1: Remote Execution

Taking inspiration from Grafana Mimir’s implementation, the ruler would be configured to send its rule query to the query-frontend component over gRPC. The querier instances receiving queries from the query-frontend (or optionally via the query-scheduler) will handle the request and send the responses to the query-frontend and be combined. The ruler will receive and process these responses as if the query had been executed locally.

Pros:

Takes full advantage of Loki’s query acceleration techniques, leading to faster and more efficient rule evaluation
Operationally simple as existing query-frontend/query-scheduler/querier setup can be used
Per-tenant isolation available in Loki’s query path (shuffle-sharding, per-tenant queues) can be used to reduce or eliminate the noisy neighbour problem

Cons:

Increased interdependence in components, increased cross-component networking
Reusing the same query-frontend/query-scheduler/querier setup can cause expensive queries to starve rule evaluations of query resources, and vice versa
- Additional complexity introduced if this setup needs to be duplicated for rule evaluations (recommended: see Other Notes section below)

Other Notes

If this feature were to be used in conjunction with rule-based sharding, this can present some further optimisation but also some additional challenges to consider.

Aside: the ruler shards by rule group by default, which means that rules can be unevenly balanced across ruler instances if some rule groups have more expensive queries than others. Another consequence of this is that rule groups execute sequentially, so expensive queries can cause subsequent rules in the group to be delayed or even missed. Rule groups are evaluated concurrently.

Rule-based sharding distributes rules evenly across all available ruler instances, each in their own rule group. Consequentially, each rule that belongs to a ruler instance will be evaluated concurrently (as they’re each in their own rule group). For tenants with hundreds or thousands of rules, this can result in large batches of queries being sent to the query-frontend in quick succession, should they all use the same interval or happen to overlap.

Assuming the remote rule evaluation takes place on the same read path that is used to execute tenant queries, care must be taken by operators who run large multi-tenant setups to ensure that large volumes of queries can be received, queued, and processed in an acceptable timeframe. The query-scheduler component is highly recommended in these situations, as it will enable the query-frontend and query components to scale out to accommodate the load. Shuffle-sharding should also be implemented to ensure that tenants with particularly large workloads do not starve out the query resources of other tenants. Alerting should also be put in place to notify operators if rule evaluations are being routinely missed or a tenants’ query queues become full.

If rule evaluations and tenant queries are slowing each other down, the read path setup would need to be duplicated so that tenant queries and rule evaluations would not share the same query execution resources.

Rule-based sharding and remote evaluation can (and should) be implemented separately. Operators should first implement remote evaluation to improve ruler reliability, and then further investigate rule-based sharding if rule evaluations are still being missed due to the sequential execution of rule groups, or advise their tenants to split these rule groups up.

0003: Query fairness across users within tenants

Wed, 06 Sep 2023 12:47:18 +0000

0003: Query fairness across users within tenants

Author: Christian Haudum (christian.haudum@grafana.com)

Date: 02/2023

Sponsor(s): @chaudum @owen-d

Type: Feature

Status: Accepted

Related issues/PRs:

Thread from mailing list:

Background

The query scheduler (or short scheduler) is a component of Loki that distributes requests (sub-queries) from the query frontend (or short frontend) to the querier workers so that execution fairness between tenants can be guaranteed.

By maintaining separate FIFO queues for each tenant and assigning the correct amount of querier workers to these queues, the scheduler takes care that a single tenant cannot compromise all other tenants’ query capabilities.

Component diagram:

Sequence diagram:

Problem Statement

Even though Loki is built as multi-tenant system by default, there are use-cases where a Loki installation only has a very large, single tenant, e.g. dedicated Loki cells for customers in Grafana Cloud.

However, there are potentially a lot of different users using the same tenant to query logs, such as users accessing Loki from Grafana or via CLI or HTTP API. This can lead to contention between queries of different users, because they all share the same tenant.

While the current implementation of the scheduler queues allows for QoS guarantees between tenants, it does not account for QoS guarantees across individual users within a single tenant.

That said, Loki does not have the notation of individual users.

Goals

The main goal of the following proposals is to lay out ideas how to improve the scheduler component to not only assure QoS across tenants, but also across actors (users) within a tenant, without requiring any changes to the deployment model of frontend, scheduler and queriers. This should also include changes to the queue structure to be easily extensible for future scheduling improvements.

Non-Goals (optional)

While changing and extending the scheduler requires also user-facing API changes, the public API is not part of the discussion of this document.

Proposals

Proposal 0: Do nothing

An alternative to changing the scheduling mechanism is to handle QoS control via multiple tenants and multi-tenant querying.

Pros:

Keeps the scheduler as simple as it is now
No development time

Cons:

While that separation into tenants may work for some prospects, it might not be feasible to implement for others.

Proposal 1: Add fixed second level to scheduler

The current scheduler is implemented in a way that it maintains a separate FIFO queue for each tenant. When a request (sub-query) is enqueued, the scheduler puts it into the existing queue for that tenant. If the queue does not exist yet, it creates it first and re-assignes the connected querier workers to the available tenant queues. Each querier worker pulls round-robin from the assigned queues in a loop.

Now, instead of enqueuing and pulling directly from the per-tenant queue, requests get enqueued in per-user queues and the per-tenant queue pulls round-robin from the user queues that are assigned to the tenant queues.

Component diagram:

Like the current implementation, the scheduler enqueues requests based on the X-Scope-OrgID header (or equivalent key in the request context), but also takes a second key (such as X-Scope-UserID) into account. This ensembles a fixed hierarchy with two levels where the tenant-to-user relation is a one-to-many relation. However, this has the disadvantage that the concept of users (that does not exist yet in Loki) leaks into the scheduler domain.

Pros:

Relatively simple to to implement

Cons:

Not extensible
Leaks domain knowledge

Proposal 2: Fully hierarchical scheduler

This proposal is similar to Proposal 1, but with the difference that there are no fixed levels and levels can be nested arbitrarily.

Component diagram:

The implementation of the RequestQueue, which controls what querier workers are connected to which root queues (aka tenant queues), can be kept as is. However, the concept of tenants and users is dropped and replaced by by a concept of hierarchical actors, which can be represented as a slice of identifiers. Note, this does not drop the concept of tenants throughout Loki (represented in the X-Scope-OrgID header and/or request context).

Example of identifiers:

actorA := []string{"tenant_a", "user_1"}
actorB := []string{"tenant_b", "user_2"}
actorC := []string{"tenant_b", "user_3", "service_foo"}
actorD := []string{"tenant_b", "user_3", "service_bar"}

More generally:

actorN := []string{"L0 Queue", "L1 Queue", "L2 Queue", ... "Ln Queue"}

The L0 queue (root queue) needs to be able to handle worker connections and therefore needs additional functionality compared to its leaf queues.

The following code snippet is meant to show the simplified recursive structure of the queues.

type Request interface{}

type Queue interface {
    Deqeue(actor []string) Request
    Enqueue(r Request, actor []string) error
}

// RequestQueue implements Queue
type RequestQueue struct {
    queriers   map[string]*querier
    rootQueues map[string]*RootQueue
}

// RootQueue implements Queue
type RootQueue struct {
    queriers map[string]*querier
    leafs    map[string]*LeafQueue
    ch       chan Request
}

// LeafQueue implements Queue
type LeafQueue struct {
    leafs map[string]*LeafQueue
    ch    chan Request
}

Pros:

Backwards compatible, because tenant can be identified as []string{"tenantID"}
Queue hierarchy can be extended without changing the scheduler implementation
Implementation does not require knowledge outside of its domain

Cons:

More complex to implement than fixed amount of levels
Each queue comes with memory overhead

Proposal 3: Multiple per-tenant sub-queues

Another option to keep the concept of users out of Loki and still provide some query fairness guarantees would be to simply shard request across multiple sub-queues within a tenant’s queue. The shard size could be a per-tenant setting to account for different tenant sizes.

This is similar to Proposal 1, in the sense of adding another fixed level of sub-queues. However, with the difference, that in this case, a single query request is assigned a random identifier that is hashed. When the query is split, the sub-requests maintain the same hashed identifier. The modulor of the hash defines to which sub-queue of a tenant requests will be enqueued.

Pros:

User agnostic per-request QoS control

Cons:

Requests of individual users can still effect other users
Not extensible

Alternative:

Sharding on a per-request basis can still be achieved with Proposal 2, by adding the request hash as an additional level in the hierarchy.

actor := []string{"tenant", "user", "request_hash"}

Consensus

Proposal 2 is going to be implemented.