Design documents on Grafana Labs

Labels

Thu, 09 Apr 2026 02:28:18 +0000

Labels

Author: Ed Welch Date: February 2019

This is the official version of this doc as of 2019/04/03, the original discussion was had via a Google doc, which is being kept for posterity but will not be updated moving forward.

Problem Statement

We should be able to filter logs by labels extracted from log content.

Keeping in mind: Loki is not a log search tool and we need to discourage the use of log labels as an attempt to recreate log search functionality. Having a label on “order number” would be bad, however, having a label on “orderType=plant” and then filtering the results on a time window with an order number would be fine. (think: grep “plant” | grep “12324134” ) Loki as a grep replacement, log tailing or log scrolling tool is highly desirable, log labels will be useful in reducing query results and improving query performance, combined with logQL to narrow down results.

Use Cases

As defined for prometheus “Use labels to differentiate the characteristics of the thing that is being measured” There are common cases where someone would want to search for all logs which had a level of “Error” or for a certain HTTP path (possibly too high cardinality), or of a certain order or event type. Examples:

Log levels.
HTTP Status codes.
Event type.

Challenges

Logs are often unstructured data, it can be very difficult to extract reliable data from some unstructured formats, often requiring the use of complicated regular expressions.
Easy to abuse. Easy to create a Label with high cardinality, even possibly by accident with a rogue regular expression.
Where do we extract metrics and labels at the client (Promtail or other?) or Loki? Extraction at the server (Loki) side has some pros/cons. Can we do both? At least with labels we could define a set of expected labels and if Loki doesn’t receive them they could be extracted.
- Server side extraction would improve interoperability at the expense of increase server workload and cost.
- Are there discoverability questions/concerns with metrics exposed via Loki vs the agent? Maybe this is better/easier to manage?
- Potentially more difficult to manage configuration with the server side having to match configs to incoming log streams

Existing Solutions

There already exist solutions for extracting processing and extracting metrics from unstructured log data, however, they will not quite work for extracting labels without some work and neither support easy inclusion as a library. It’s worth noting and understanding how they work to try to get the best features in our solution. mtail https://github.com/google/mtail 1721 github stars, huge number of commits, releases and contributors, google project

All go, uses go RE2 regular expressions which is going to be more performant than grok_exporter below which uses a full regex implementation allowing backtracking and lookahead required to be compliant with Grok but which are also slower.

grok_exporter https://github.com/fstab/grok_exporter 278 github stars, mature/active project

If you are familiar with Grok this would be more comfortable, many people use ELK stacks and would likely be familiar with or already have Grok strings for their logs, making it easy to use grok_exporter to extract metrics.

One caveat is the dependency on the oniguruma C library which parses the regular expressions.

Implementation

Details

As mentioned previously in the challenges for working with unstructured data, there isn’t a good one size fits all solution for extracting structured data.

The Docker log format is an example where multiple levels of processing may be required, where the docker log is json, however, it also contains the log message field which itself could be embedded json, or a log message which needs regex parsing.

A pipelined approach should allow for handling these more challenging scenarios

There are 2 interfaces within Promtail already that should support constructing a pipeline:

type EntryMiddleware interface {
    Wrap(next EntryHandler) EntryHandler
}

type EntryHandler interface {
    Handle(labels model.LabelSet, time time.Time, entry string) error
}

Essentially every entry in the pipeline will Wrap the log line with another EntryHandler which can add to the LabelSet, set the timestamp, and mutate(or not) the log line) before it gets handed to the next stage in the pipeline.

Example

{
  "log": "level=info msg=\”some log message\”\n",
  "stream": "stderr",
  "time": "2012-11-01T22:08:41+00:00"
}

This is a docker format log file which is JSON but also contains a log message which has some key-value pairs.

Our pipelined config might look like this:

scrape_configs:
- job_name: system
  pipeline_stages:
    - json:
        timestamp:
          source: time
          format: RFC3339
        labels:
          stream:
            source: json_key_name.json_sub_key_name
        output: log
    - regex:
        expr: '.*level=(?P<level>[a-zA-Z]+).*'
        labels:
          level:
    - regex:
        expr: '.*msg=(?P<message>[a-zA-Z]+).*'
        output: message

Looking at this a little closer:

     - json:
        timestamp:
          source: time
          format: TODO                               ①
        labels:
          stream:
            source: json_key_name.json_sub_key_name  ②
        output: log                                  ③

① The format key will likely be a format string for Go’s time.Parse or a format string for strptime, this still needs to be decided, but the idea would be to specify a format string used to extract the timestamp data, for the regex parser there would also need to be a expr key used to extract the timestamp. ② One of the json elements was “stream” so we extract that as a label, if the json value matches the desired label name it should only be required to specify the label name as a key, if some mapping is required you can optionally provide a “source” key to specify where to find the label in the document. (Note the use of json_key_name.json_sub_key_name is just an example here and doesn’t match our example log) ③ Tell the pipeline which element from the json to send to the next stage.

    - regex:
        expr: '.*level=(?P<level>[a-zA-Z]+).*'  ①
        labels:
          level:                                ②

① Define the Go RE2 regex, making sure to use a named capture group. ② Extract labels using the named capture group names.

Notice there was not an output section defined here, omitting the output key should instruct the parser to return the incoming log message to the next stage with no changes.

    - regex:
        expr: '.*msg=(?P<message>[a-zA-Z]+).*'
        output: message                          ①

① Send the log message as the output to the last stage in the pipeline, this will be what you want Loki to store as the log message.

There is an alternative configuration that could be used here to accomplish the same result:

scrape_configs:
- job_name: system
  pipeline_stages:
    - json:
        timestamp:
          source: time
          format: FIXME
        labels:
          stream:
        output: log
    - regex:
        expr: '.*level=(?P<level>[a-zA-Z]+).*msg=(?P<message>[a-zA-Z]+).*'
        labels:
          level:                                                             ①
          log:
            source: message                                                  ②
        output: message

① Similar to the json parser, if your log label matches the regex named group, you need only specify the label name as a yaml key ② If you had a use case for specifying a different label name from the regex group name you can optionally provide the source key with the value matching the named capture group.

You can define a more complicated regular expression with multiple capture groups to extract many labels and/or the output log message in one entry parser. This has the advantage of being more performant, however, the regular expression will also get much more complicated.

Note the regex for message is incomplete and would do a terrible job of matching any standard log message which might contain spaces or non alpha characters.

Concerns

Debugging, especially if a pipeline stage is mutating the log entry.
Clashing labels and how to handle this (two stages try to set the same label)
Performance vs ease of writing/use, if every label is extracted one at a time and there are a lot of labels and a long line, it would force reading the line many times, however contrast this to a really long complicated regex which only has to read the line once but is difficult to write and/or change and maintain

Further improvements

There are some basic building blocks for our pipeline which will use the EntryMiddleware interface, the two most commonly used will likely be:

Regex Parser
JSON Parser

However we don’t want to ask people to copy and paste basic configs over and over for very common use cases, so it would make sense to add some additional parsers which would really be supersets of the base parsers above.

For example, the config above might be simplified to:

scrape_configs:
- job_name: system
  pipeline_stages:
    - docker:

scrape_configs:
- job_name: system
  pipeline_stages:
    - cri:

Which could still easily be extended to extract additional labels:

scrape_configs:
- job_name: system
  pipeline_stages:
    - docker:
    - regex:
        expr: '.*level=(?P<level>[a-zA-Z]+).*'
        labels:
          level:

Auto Detection?

An even further simplification would be to attempt to autodetect the log format, a PR for this work has been submitted, then the config could be as simple as:

scrape_configs:
- job_name: system
  pipeline_stages:
    - auto:

This certainly has some advantages for people first adopting and testing Loki, allowing them to point it at their logs and at least get the timestamp and log message extracted properly for the common formats like Docker and CRI.

There are also some challenges with auto detection and edge cases, though most people are going to want to augment the basic config with additional labels, so maybe it makes sense to default to auto but suggest when people start writing configs they chose the correct parser?

Other Thoughts and Considerations

We should have a standalone client in some fashion which allows for testing of log parsing at the command line, allowing users to validate regular expressions or configurations to see what information is extracted.
Other input formats to the pipeline which are not reading from log files, such as containerd grpc api, or from stdin or unix pipes, etc.
It would be nice if at some point we could support loading code into the pipeline stages for even more advanced/powerful parsing capabilities.

Promtail Push API

Thu, 09 Apr 2026 02:28:18 +0000

Promtail Push API

Author: Robert Fratto (@rfratto)
Date: Feb 4 2020
Status: DRAFT

Despite being an optional piece of software, Promtail provides half the power of Loki’s story: log transformations, service discovery, metrics from logs, and context switching between your existing metrics and logs. Today, Promtail can only be operated to consume logs from very specific sources: files, journal, or syslog. If users wanted to write custom tooling to ship logs, the tooling has to bypass Promtail and push directly to Loki. This can lead users to reimplement functionality Promtail already provides, including its error retries and batching code.

This document proposes a Push API for Promtail. The preferred implementation is by copying the existing Loki Push API and implementing it for Promtail. By being compatible with the Loki Push API, the Promtail Push API can allow batches of logs to be processed at once for optimize performance. Matching the Promtail API also allows users to transparently switch their push URLs from their existing tooling. Finally, a series of alternative solutions are detailed.

Configuration

Promtail will have a new target called HTTPTarget, configurable in the scrape_config array with the following schema:

# Defines an HTTP target, which exposes an endpoint against the Promtail
# HTTP server to accept log traffic.
http:
  # Defines the base URL for the push path, adding a prefix to the
  # exposed endpoint. The final endpoint path is
  # <base_url>loki/api/v1/push. If omitted, defaults to /.
  #
  # Multiple http targets with the same base_url must not exist.
  base_url: /

  # Map of labels to add to every log line passed through to the target.
  labels: {}

Considerations

Users will be able to define multiple http scrape configs, but the base URL value must be different for each instance. This allows to cleanly separate pipelines through different push endpoints.

Users must also be aware about problems with running Promtail with an HTTP target behind a load balancer: if payloads are load balanced between multiple Promtail instances, ordering of logs in Loki will be disrupted leading to rejected pushes. Users are recommended to do one of the following:

Have a dedicated Promtail instance for receiving pushes. This also applies to using the syslog target.
Have a separate Kubernetes service that always resolves to the same Promtail pod, bypassing the load balancing issue.

Implementation

As discussed in this document, this feature will be implemented by copying the existing Loki Push API and exposing it via Promtail.

Considered Alternatives

Using the existing API was chosen for its simplicity and capabilities of being used for interesting configurations (e.g., chaining Promtails together). These other options were considered but rejected as not the best solution for the problem being solved.

Note that Option 3 has value and may be implemented separately from this feature.

Option 1: JSON / Protobuf Payload

A new JSON and Protobuf payload format can be designed instead of the existing Loki push payload. Both formats would have to be exposed to support clients that either can’t or won’t use protobuf marshalling.

The primary benefit of this approach is to allow us to tweak the payload schema independently of Loki’s existing schema, but otherwise may not be very useful and is essentially just code duplication.

Option 2: gRPC Service

The logproto.Pusher service could be exposed through Promtail. This would enable clients stubs to be generated for languages that have gRPC support, and, for HTTP1 support, a gRPC gateway would be embedded in Promtail itself.

This implementation option is similar to the original proposed solution, but uses the gRPC gateway to handle HTTP/1 traffic instead of the HTTP1 shim that Loki uses. There are some concerns with this approach:

The gRPC Gateway reverse proxy will need to play nice with the existing HTTP mux used in Promtail.
We couldn’t control the HTTP and Protobuf formats separately as Loki can.
Log lines will be double-encoded thanks to the reverse proxy.
A small overhead of using a reverse proxy in-process will be introduced.
This breaks our normal pattern of writing our own shim functions; may add some cognitive overhead of having to deal with the gRPC gateway as an outlier in the code.

Option 3: Plaintext Payload

Prometheus’ Push Gateway API is cleverly designed and we should consider implementing our API in the same format: users would push to http://promtail-url/push/label1/value1?timestamp=now with a plaintext POST body. For example:

curl -X POST http://promtail.default/push/foo/bar/fizz/buzz -d “hello, world!”

This approach may be slightly faster when compared to a non-plaintext payload as no unmarshaling needs to be performed. This URL path and timestamp still needs to be parsed, but this will generally be faster than the reflection requirements imposed by JSON.

However, note that this API limits Promtail to accepting one line at a time and may cause performance issues when trying to handle large volumes of traffic. As an alternative, this API could also be implemented by external tooling and be built on top of any of the other implementation options.

An example implementation was created and has received positive support for its simplicity and ease of integration.

Write-Ahead Logs

Thu, 09 Apr 2026 02:28:18 +0000

Write-Ahead Logs

Author: Owen Diehl - owen-d (Grafana Labs)

Date: 30/09/2020

Impetus

Loki already takes numerous steps to ensure the persistence of log data, most notably the use of a configurable replication factor (redundancy) in the ingesters. However, this still leaves much to be desired in persistence guarantees, especially for single binary deployments. This proposal outlines a write ahead log (WAL) in order to complement existing measures by allowing storage/replay of incoming writes via local disk on the ingester components.

Strategy

We suggest a two pass WAL implementation which includes an initial recording of accepted writes (segments) and a subsequent checkpointing (checkpoints) which coalesces the first pass into more efficient representations to speed up replaying.

Segments

Segments are the first pass and most basic WAL. They store individual records of incoming writes that have been accepted and can be used to reconstruct the in memory state of an ingester without any external input. Each segment is some multiple of 32kB and upon filling one segment, a new segment is created. Initially Loki will try 256kb segment sizes, readjusting as necessary. They are sequentially named on disk and are automatically created when a target size is hit as follows:

data
└── wal
   ├── 000000
   ├── 000001
   └── 000002

Truncation

In order to prevent unbounded growth and remove operations which have been flushed to storage from the WAL, it is regularly truncated and all but the last segment (which is currently active) are deleted at a configurable interval (ingester.checkpoint-duration). This is where checkpoints come into the picture.

Checkpoints

Before truncating the WAL, we advance the WAL segments by one in order to ensure we don’t delete the currently writing segment. The directory will look like:

data
└── wal
   ├── 000000
   ├── 000001
   ├── 000002 <- likely not full, no matter
   └── 000003 <- newly written, empty

Each in memory stream is iterated across an interval, calculated by checkpoint_duration / in_memory_streams and written to the checkpoint. After the checkpoint completes, it is moved from its temp directory to the ingester.wal-dir, taking the name of the last segment before it started (checkpoint.000002) and then all applicable segments (00000, 00001, 00002) and any previous checkpoint are deleted.

Afterwards, it will look like:

data
└── wal
   ├── checkpoint.000002 <- completed checkpoint
   └── 000003 <- currently active wal segment

Queueing Checkpoint operations

It’s possible that one checkpoint operation will start at the same time another is running. In this case, the existing checkpoint operation should disregard its internal ticker and flush its series as fast as possible. Afterwards, the next checkpoint operation can begin. This will likely create a localized spike in IOPS before the amortization of the following checkpoint operation takes over and is another important reason to run the WAL on an isolated disk in order to mitigate noisy neighbor problems. After we’ve written/moved the current checkpoint, we reap the old one.

WAL Record Types

Streams

A Stream record type is written when an ingester receives a push for a series it doesn’t yet have in memory. At a high level, this will contain

type SeriesRecord struct {
	UserID  string
	Labels labels.Labels
	Fingerprint uint64 // label fingerprint
}

Logs

A Logs record type is written when an ingester receives a push, containing the fingerprint of the series it refers to and a list of (timestamp, log_line) tuples, after a Stream record type is written, if applicable.

type LogsRecord struct {
	UserID  string
	Fingperprint uint64 // label fingerprint for the series these logs refer to
	Entries []logproto.Entry
}

Restoration

Replaying a WAL is done by loading any available checkpoints into memory and then replaying any operations from successively named segments on top (checkpoint.000003 -> 000004 -> 000005, etc). It’s likely some of these operations will fail because they’re already included in the checkpoint (due to delay introduced in our amortizations), but this is ok – we won’t lose any data, only try to write some data twice, which will be ignored.

Deployment

Introduction of the WAL requires that ingesters have persistent disks which are reconnected across restarts (this is a good fit for StatefulSets in Kubernetes). Additionally, it’s recommended that the WAL uses an independent disk such that it’s isolated from being affected by or causing noisy neighbor problems, especially during any IOPS spike(s).

Implementation goals

Use underlying prometheus wal pkg when possible for consistency and to mitigate undifferentiated heavy lifting. Interfaces handle page alignment and use []byte.
- Ensure this package handles arbitrarily long records (log lines in Loki’s case).
Ensure our in memory representations can be efficiently moved to/from []byte in order to generate conversions for fast/efficient loading from checkpoints.
Ensure chunks which have already been flushed to storage are kept around for ingester.retain-period, even after a WAL replay.

Alternatives

Use the Cortex WAL

Since we’re not checkpointing from the WAL records but instead doing a memory dump, this isn’t bottlenecked by throughput but rather memory size. Therefore we can start with checkpointing by duration rather than accounting for throughput as well. This makes the proposed solution nearly identical to the Cortex WAL approach. The one caveat is that wal segments will accrue between checkpoint operations and may constitute a large amount of data (log throughput varies). We may eventually consider other routes to handle this if duration based checkpointing proves insufficient.

Don’t build checkpoints from memory, instead write new WAL elements

Instead of building checkpoints from memory, this would build the same efficiencies into two distinct WAL Record types: Blocks and FlushedChunks. The former is a record type which will contain an entire compressed block after it’s cut and the latter will contain an entire chunk + the sequence of blocks it holds when it’s flushed. This may offer good enough amortization of writes because block cuts are assumed to be evenly distributed and chunk flushes have the same property and use jitter for synchronization.

This could be used to drop WAL records which have already elapsed the ingester.retain-period, allowing for faster WAL replays and more efficient loading.

type FlushRecord struct {
  Fingerprint uint64 // labels
  FlushedAt uint64 // timestamp when it was flushed, can be used with `ingester.retain-period` to either keep or discard records on replay
  LastEntry logproto.Entry // last entry included in the flushed chunk
}

It would also allow building checkpoints without relying on an ingester’s internal state, but would likely require multiple WALs, partitioned by record type in order to be able to iterate all FlushedChunks -> Blocks -> Series -> Samples such that we could no-op the later (lesser priority) types that are superseded by the former types. The benefits do not seem worth the cost here, especially considering the simpler suggested alternative and the extensibility costs if we need to add new record types if/when the ingester changes internally.

Ordering Constraint Removal

Thu, 09 Apr 2026 02:28:18 +0000

Ordering Constraint Removal

Author: Owen Diehl - owen-d (Grafana Labs)

Date: 28/01/2021

Problem

Loki imposes an ordering constraint on ingested data; that is to say incoming data must have monotonically increasing timestamps, partitioned by stream. This has historical inertia from our parent project, Cortex, but presents unintended consequences specific to log ingestion. In contrast to metric scraping, Loki has reasonable use cases where the ordering constraint poses a problem, including:

Ingesting logs from a cloud function without feeling pressured to add high cardinality labels like invocation_id to avoid out of order errors.
Ingesting logs from other agents/mechanisms that don’t take into account Loki’s ordering constraint. For instance, fluent{d,bit} variants may batch and retry writes independently of other batches, causing unpredictable log loss via out of order errors.

Many of these illustrate the adversity between ordering and cardinality. In addition to enabling some previously difficult/impossible use cases, removing the ordering constraint lets us avoid potential conflict between these two concepts and helps incentivize good practice in the form of fewer useful labels.

Requirements

Enable out of order writes
Maintain query interface parity

Bonuses

Optimize for in order writes

Alternatives

Implement order-agnostic blocks + increase memory usage by compression_ratio: Deemed unacceptable due to TCO (total cost of ownership).
Implement order-agnostic blocks + scale horizontally (reduce per-ingester streams): Deemed unacceptable due to TCO and increasing ring pressure.
Implement order-agnostic blocks + flush chunks more frequently: Deemed unacceptable due to negatively increasing the index and the number of chunks requiring merge during reads.

Design

Background

I suggest allowing a stream’s head block to accept unordered writes and later re-order cut blocks similar to merge-sort before flushing them to storage. Currently, writes are accepted in monotonically increasing timestamp order to a headBlock, which is occasionally “cut” into a compressed, immutable block. In turn, these blocks are combined into a chunk and persisted to storage.

Figure 1

    Data while being buffered in Ingester          |                                Chunk in storage
                                                   |
    Blocks                    Head                 |       ---------------------------------------------------------------------
                                                   |       |   ts0   ts1    ts2   ts3    ts4   ts5    ts6   ts7    ts8    ts9  |
--------------           ----------------          |       |   ---------    ---------    ---------    ---------    ---------   |
|    blocks  |--         |  head block  |          |       |   |block 0|    |block 1|    |block 2|    |block 4|    |block 5|   |
|(compressed)| |         |(uncompressed)|          |       |   |       |    |       |    |       |    |       |    |       |   |
|            | | ------> |              |          |       |   ---------    ---------    ---------    ---------    ---------   |
|            | |         |              |          |       |                                                                   |
-------------- |         ----------------          |       ---------------------------------------------------------------------
  |            |                                   |
  --------------                                   |

Historically because of Loki’s ordering constraint, these blocks maintain a monotonically increasing timestamp (abbreviated ts) order where

Figure 2

start       end
ts0         ts1          ts2        ts3
--------------           --------------
|            |           |            |
|            | --------> |            |
|            |           |            |
|            |           |            |
--------------           --------------

This allows us two optimzations:

We can store much more data in memory because each block is compressed after being cut from a head block.
We can query the block’s metadata, such as ts0 and ts1 and skip querying it in the case of i.e. the timestamps are outside a request’s bounds.

Unordered Head Blocks

The head block’s internal structure will be replaced with a tree structure, enabling logarithmic inserts/lookups and n log(n) scans. Cutting a block from the head block will iterate through this tree, creating a sorted block identical to the ones currently in use. However, because we’ll be accepting arbitrarily-ordered writes, there will no longer be any guaranteed inter-block order. In contrast to figure 2, blocks may have overlapping data:

Figure 3

start       end
ts1         ts3          ts0        ts2
--------------           --------------
|            |           |            |
|            | --------> |            |
|            |           |            |
|            |           |            |
--------------           --------------

Thus all blocks must have their metadata checked against a query. In this example, a query for the bounds [ts1,ts2] would need to decompress and scan the [ts1, ts2] range across both of them, but a query against [ts3, ts4] would only decompress and scan one block.

Figure 4

     chunk1
-------------------
                chunk2
         ---------------------
         query range requiring both
         ----------
                             query range requiring chunk2 only
                             -----------
ts0     ts1      ts2       ts3        ts4 (not in any block)
------------------------------
|        |        |          |
|        |        |          |
|        |        |          |
|        |        |          |
------------------------------

The performance losses against the current approach includes:

Appending a log line is now performed in logarithmic time instead of amortized (due to array resizing) constant time.
Blocks may contain overlapping data (although ordering is still guaranteed within each block).
Head block scans are now O(n log(n)) instead of O(n)

Flushing and Chunk Creation

Loki regularly combines multiple blocks into a chunk and “flushes” it to storage. In order to ensure that reads over flushed chunks remain as performant as possible, we will re-order a possibly-overlapping set of blocks into a set of blocks that maintain monotonically increasing order between them. From the perspective of the rest of Loki’s components (queriers/rulers fetching chunks from storage), nothing has changed.

Note
In the case that data for a stream is ingested in order, this is effectively a no-op, making it well optimized for in-order writes (which is both the requirement and default in Loki currently). Thus, this should have little performance impact on ordered data while enabling Loki to ingest unordered data.

Chunk Durations

When --validation.reject-old-samples is enabled, Loki accepts incoming timestamps within the range

[now() - `--validation.reject-old-samples.max-age`, now() + `--validation.create-grace-period`]

For most of our clusters, this would mean the range of acceptable data is one week long. In contrast, our max chunk age is 2h. Allowing unordered writes would mean that ingesters would willingly receive data for 168h, or up to 84 distinct chunk lengths. This presents a problem: a malicious user could be writing to many (84 in this case) distinct chunks simultaneously, flooding Loki with underutilized chunks which bloat the index.

In order to mitigate this, there are a few options (not mutually exclusive):

Lower the valid acceptance range
Create an active validity window, such as [most_recent_sample-max_chunk_age, now() + creation_grace_period].

The first option is simple, already available, and likely somewhat reasonable. The second is simple to implement and an effective way to ensure Loki can ingest unordered logs but maintain a sliding validity window. I expect this to cover nearly all reasonable use cases and effectively mitigate bad actors.

Chunk Synchronization

We also cut chunks according to the sync_period. The first timestamp ingested past this bound will trigger a cut. This process aids in increasing chunk determinism and therefore our deduplication ratio in object storage because chunks are content addressed. With the removal of our ordering constraint, it’s possible that in some cases the synchronization method will not be as effective, such as during concurrent writes to the same stream across this bound.

Note
It’s important to mention that this is possible today with the current ordering constraint, but we’ll be increasing the likelihood by removing it.

Figure 5

       Concurrent Writes over threshold
                   ^ ^
                   | |
                   | |
-----------------|-----------------
                 |
                 v
             Sync Marker

To mitigate this problem and preserve the benefits of chunk deduplication, we’ll need to make chunk synchronization less susceptible to non-determinism during concurrent writes. To do this, we can move the synchronization trigger from the Append code path to the asynchronous FlushLoop. Note, the semantics for when a chunk is cut will not change: that is, on the first timestamp crossing the synchronization bound. However, cutting the chunks for synchronization on the flush path mitigates the likelihood of different chunks being cut. In order to cut multiple chunks with different hashes, appends would then need to cross this boundary at the same time the flush loop checks the stream, which should be very unlikely.

Future Opportunities

This ends the initial design portion of this document. Below, I’ll describe some possible changes we can address in the future, should they become warranted.

Variance Budget

The intended approach of a “sliding validity” window for each stream is simple and effective at preventing misuse and bad actors from writing across the entire acceptable range for incoming timestamps. However, we may in the future wish to take a more sophisticated approach, introducing per tenant “variance” budgets, likely derived from the stream limit. This ingester limit could, for example use an incremental (online) standard deviation/variance algorithm such as Welford’s, which would allow writing to larger ranges than option (2) in the Chunk Durations section.

LSM Tree

Much of the proposed approach mirrors an LSM-Tree (Log Structured Merge Tree), albeit in memory instead of using disk. What a weird choice – LSM Trees are designed to effectively use disk, so why not go that route? We currently have no wish to add extra disk dependencies to Loki where we can avoid it, but below I will outline what an LSM-Tree approach would look like. Ultimately, using disk would enable buffering more data in the ingester before flushing,

Allowing us to

Flush more efficiently utilized chunks (in some cases)
Keep open a wider validity window for incoming logs

At the cost of

being susceptible to disk-related complexity and problems

MemTable (head block)

Writes in an LSM-Tree are first accepted to an in-memory structure called a memtable (generally a balancing tree such as red-black) until the memtable hits a preconfigured size. In Loki, this corresponds to the stream’s head block, which is uncompressed.

SSTables (blocks)

Once a Memtable (head block) in an LSM-Tree hits a predefined size, it is flushed to disk as an immutable sorted structure called an SSTable (sorted strings table). In Loki, we can use either the pre-existing MemChunk format, which is ordered, compact, and contains a block index within it, or the pre-existing block format directly. These are stored on disk to lessen memory pressure and loaded for queries when necessary.

Block Index

Incoming reads in an LSM-Tree may need access to the SSTable entries in addition to the currently active memtable (head block). In order to improve this, we may cache the metadata including block offsets, start and end timestamps within an SSTable (block || MemChunk) in memory to mitigate lookups, seeking, and loading unnecessary data from disk.

Compaction (flushing)

Compaction in an LSM-Tree combines and reorders multiple SSTables (blocks || MemChunks). This is mainly covered in the Flushing section of the in-memory approach, but compaction is equivalent to flushing for our case. That is, merge multiple SSTables on disk together in an algorithm reminiscent of merge sort and flush them to storage in our ordered chunk format.