Get started with Grafana Loki on Grafana Labs

Loki overview

Thu, 09 Apr 2026 02:28:18 +0000

Loki overview

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. Loki differs from Prometheus by focusing on logs instead of metrics, and collecting logs via push, instead of pull.

Loki is designed to be very cost effective and highly scalable. Unlike other logging systems, Loki does not index the contents of the logs, but only indexes metadata about your logs as a set of labels for each log stream.

A log stream is a set of logs which share the same labels. Labels help Loki to find a log stream within your data store, so having a quality set of labels is key to efficient query execution.

Log data is then compressed and stored in chunks in an object store such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even, for development or proof of concept, on the filesystem. A small index and highly compressed chunks simplify the operation and significantly lower the cost of Loki.

A typical Loki-based logging stack consists of 3 components:

Agent - An agent or client, for example Grafana Alloy. The agent scrapes logs, turns the logs into streams by adding labels, and pushes the streams to Loki through an HTTP API.
Loki - The main server, responsible for ingesting and storing logs and processing queries. It can be deployed in three different configurations, for more information see deployment modes.
Grafana for querying and displaying log data. You can also query logs from the command line, using LogCLI or using the Loki API directly.

Loki features

Scalability - Loki is designed for scalability, and can scale from as small as running on a Raspberry Pi to ingesting petabytes a day. In its most common deployment, “simple scalable mode”, Loki decouples requests into separate read and write paths, so that you can independently scale them, which leads to flexible large-scale installations that can quickly adapt to meet your workload at any given time. If needed, each of the Loki components can also be run as microservices designed to run natively within Kubernetes.
Multi-tenancy - Loki allows multiple tenants to share a single Loki instance. With multi-tenancy, the data and requests of each tenant is completely isolated from the others. Multi-tenancy is configured by assigning a tenant ID in the agent.
Third-party integrations - Several third-party agents (clients) have support for Loki, via plugins. This lets you keep your existing observability setup while also shipping logs to Loki.
Efficient storage - Loki stores log data in highly compressed chunks. Similarly, the Loki index, because it indexes only the set of labels, is significantly smaller than other log aggregation tools. By leveraging object storage as the only data storage mechanism, Loki inherits the reliability and stability of the underlying object store. It also capitalizes on both the cost efficiency and operational simplicity of object storage over other storage mechanisms like locally attached solid state drives (SSD) and hard disk drives (HDD).
The compressed chunks, smaller index, and use of low-cost object storage, make Loki less expensive to operate.
LogQL, the Loki query language - LogQL is the query language for Loki. Users who are already familiar with the Prometheus query language, PromQL, will find LogQL familiar and flexible for generating queries against the logs. The language also facilitates the generation of metrics from log data, a powerful feature that goes well beyond log aggregation.
Alerting - Loki includes a component called the ruler, which can continually evaluate queries against your logs, and perform an action based on the result. This allows you to monitor your logs for anomalies or events. Loki integrates with Prometheus Alertmanager, or the alert manager within Grafana.
Grafana integration - Loki integrates with Grafana, Mimir, and Tempo, providing a complete observability stack, and seamless correlation between logs, metrics and traces.

Quick Start Loki

Thu, 09 Apr 2026 02:28:18 +0000

Quick Start Loki

This section provides a collection of tutorials to help you get started with Loki. Our recommenation is to do the tutorials in this order:

Loki architecture

Thu, 09 Apr 2026 02:28:18 +0000

Loki architecture

Grafana Loki has a microservices-based architecture and is designed to run as a horizontally scalable, distributed system. The system has multiple components that can run separately and in parallel. The Grafana Loki design compiles the code for all components into a single binary or Docker image. The -target command-line flag controls which component(s) that binary will behave as.

To get started easily, run Grafana Loki in “single binary” mode with all components running simultaneously in one process, or in “simple scalable deployment” mode, which groups components into read, write, and backend parts.

Grafana Loki is designed to easily redeploy a cluster under a different mode as your needs change, with no configuration changes or minimal configuration changes.

For more information, refer to Deployment modes and Components.

Storage

Loki stores all data in a single object storage backend, such as Amazon Simple Storage Service (S3), Google Cloud Storage (GCS), Azure Blob Storage, among others. This mode uses an adapter called index shipper (or short shipper) to store index (TSDB or BoltDB) files the same way we store chunk files in object storage. This mode of operation became generally available with Loki 2.0 and is fast, cost-effective, and simple. It is where all current and future development lies.

Prior to 2.0, Loki had different storage backends for indexes and chunks. For more information, refer to Legacy storage.

Data format

Grafana Loki has two main file types: index and chunks.

The index is a table of contents of where to find logs for a specific set of labels.
The chunk is a container for log entries for a specific set of labels.

The diagram above shows the high-level overview of the data that is stored in the chunk and data that is stored in the index.

Index format

There are two index formats that are currently supported as single store with index shipper:

TSDB (recommended)

Time Series Database (or short TSDB) is an index format originally developed by the maintainers of Prometheus for time series (metric) data.

It is extensible and has many advantages over the deprecated BoltDB index. New storage features in Loki are solely available when using TSDB.
BoltDB (deprecated)

Bolt is a low-level, transactional key-value store written in Go.

Chunk format

A chunk is a container for log lines of a stream (unique set of labels) of a specific time range.

The following ASCII diagram describes the chunk format in detail.

----------------------------------------------------------------------------
|                        |                       |                         |
|     MagicNumber(4b)    |     version(1b)       |      encoding (1b)      |
|                        |                       |                         |
----------------------------------------------------------------------------
|                      #structuredMetadata (uvarint)                       |
----------------------------------------------------------------------------
|      len(label-1) (uvarint)      |          label-1 (bytes)              |
----------------------------------------------------------------------------
|      len(label-2) (uvarint)      |          label-2 (bytes)              |
----------------------------------------------------------------------------
|      len(label-n) (uvarint)      |          label-n (bytes)              |
----------------------------------------------------------------------------
|                      checksum(from #structuredMetadata)                  |
----------------------------------------------------------------------------
|           block-1 bytes          |           checksum (4b)               |
----------------------------------------------------------------------------
|           block-2 bytes          |           checksum (4b)               |
----------------------------------------------------------------------------
|           block-n bytes          |           checksum (4b)               |
----------------------------------------------------------------------------
|                           #blocks (uvarint)                              |
----------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
----------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
----------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
----------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint)  | offset, len (uvarint)         |
----------------------------------------------------------------------------
|                          checksum(from #blocks)                          |
----------------------------------------------------------------------------
| #structuredMetadata len (uvarint) | #structuredMetadata offset (uvarint) |
----------------------------------------------------------------------------
|     #blocks len (uvarint)         |       #blocks offset (uvarint)       |
----------------------------------------------------------------------------

mint and maxt describe the minimum and maximum Unix nanosecond timestamp, respectively.

The structuredMetadata section stores non-repeated strings. It is used to store label names and label values from structured metadata. Note that the labels strings and lengths within the structuredMetadata section are stored compressed.

Block format

A block is comprised of a series of entries, each of which is an individual log line. Note that the bytes of a block are stored compressed. The following is their form when uncompressed:

-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-1 bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-2 bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-3 bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------
|  ts (varint)  |  len (uvarint)  |  log-n bytes  |  len(from #symbols)  |  #symbols (uvarint)  |  symbol-1 (uvarint)  | symbol-n*2 (uvarint) |
-----------------------------------------------------------------------------------------------------------------------------------------------

ts is the Unix nanosecond timestamp of the logs, while len is the length in bytes of the log entry.

Symbols store references to the actual strings containing label names and values in the structuredMetadata section of the chunk.

Write path

On a high level, the write path in Loki works as follows:

The distributor receives an HTTP POST request with streams and log lines.
The distributor hashes each stream contained in the request so it can determine the ingester instance to which it needs to be sent based on the information from the consistent hash ring.
The distributor sends each stream to the appropriate ingester and its replicas (based on the configured replication factor).
The ingester receives the stream with log lines and creates a chunk or appends to an existing chunk for the stream’s data. A chunk is unique per tenant and per label set.
The ingester acknowledges the write.
The distributor waits for a majority (quorum) of the ingesters to acknowledge their writes.
The distributor responds with a success (2xx status code) in case it received at least a quorum of acknowledged writes. or with an error (4xx or 5xx status code) in case write operations failed.

Refer to Components for a more detailed description of the components involved in the write path.

Read path

On a high level, the read path in Loki works as follows:

The query frontend receives an HTTP GET request with a LogQL query.
The query frontend splits the query into sub-queries and passes them to the query scheduler.
The querier pulls sub-queries from the scheduler.
The querier passes the query to all ingesters for in-memory data.
The ingesters return in-memory data matching the query, if any.
The querier lazily loads data from the backing store and runs the query against it if ingesters returned no or insufficient data.
The querier iterates over all received data and deduplicates, returning the result of the sub-query to the query frontend.
The query frontend waits for all sub-queries of a query to be finished and returned by the queriers.
The query frontend merges the individual results into a final result and return it to the client.

Refer to Components for a more detailed description of the components involved in the read path.

Multi-tenancy

All data, both in memory and in long-term storage, may be partitioned by a tenant ID, pulled from the X-Scope-OrgID HTTP header in the request when Grafana Loki is running in multi-tenant mode. When Loki is not in multi-tenant mode, the header is ignored and the tenant ID is set to fake, which will appear in the index and in stored chunks.

Loki components

Thu, 09 Apr 2026 02:28:18 +0000

Loki components

Loki is a modular system that contains many components that can either be run together (in “single binary” mode with target all), in logical groups (in “simple scalable deployment” mode with targets read, write, backend), or individually (in “microservice” mode). For more information see Deployment modes.

Component	individual	`all`	`read`	`write`	`backend`
Distributor	x	x		x
Ingester	x	x		x
Query Frontend	x	x	x
Query Scheduler	x	x			x
Querier	x	x	x
Index Gateway	x				x
Compactor	x	x			x
Ruler	x	x			x
Pattern ingester	x	x		x
Bloom Planner (Experimental)	x				x
Bloom Builder (Experimental)	x				x
Bloom Gateway (Experimental)	x				x

This page describes the responsibilities of each of these components.

Distributor

The distributor service is responsible for handling incoming push requests from clients. It’s the first step in the write path for log data. Once the distributor receives a set of streams in an HTTP request, each stream is validated for correctness and to ensure that it is within the configured tenant (or global) limits. Each valid stream is then sent to n ingesters in parallel, where n is the replication factor for data. The distributor determines the ingesters to which it sends a stream to using consistent hashing.

A load balancer must sit in front of the distributor to properly balance incoming traffic to them. In Kubernetes, the service load balancer provides this service.

The distributor is a stateless component. This makes it easy to scale and offload as much work as possible from the ingesters, which are the most critical component on the write path. The ability to independently scale these validation operations means that Loki can also protect itself against denial of service attacks that could otherwise overload the ingesters. It also allows us to fan-out writes according to the replication factor.

Validation

The first step the distributor takes is to ensure that all incoming data is according to specification. This includes things like checking that the labels are valid Prometheus labels as well as ensuring the timestamps aren’t too old or too new or the log lines aren’t too long.

Preprocessing

Currently, the only way the distributor mutates incoming data is by normalizing labels. What this means is making {foo="bar", bazz="buzz"} equivalent to {bazz="buzz", foo="bar"}, or in other words, sorting the labels. This allows Loki to cache and hash them deterministically.

Rate limiting

The distributor can also rate-limit incoming logs based on the maximum data ingest rate per tenant. It does this by checking a per-tenant limit and dividing it by the current number of distributors. This allows the rate limit to be specified per tenant at the cluster level and enables us to scale the distributors up or down and have the per-distributor limit adjust accordingly. For instance, say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/s before limiting. Now, say another large tenant joins the cluster and we need to spin up 10 more distributors. The now 20 distributors will adjust their rate limits for tenant A to (10MB / 20 distributors) = 500KB/s. This is how global limits allow much simpler and safer operation of the Loki cluster.

Note
The distributor uses the ring component under the hood to register itself amongst its peers and get the total number of active distributors. This is a different “key” than the ingesters use in the ring and comes from the distributor’s own ring configuration.

Forwarding

Once the distributor has performed all of its validation duties, it forwards data to the ingester component which is ultimately responsible for acknowledging the write operation.

Replication factor

In order to mitigate the chance of losing data on any single ingester, the distributor will forward writes to a replication factor of them. Generally, the replication factor is 3. Replication allows for ingester restarts and rollouts without failing writes and adds additional protection from data loss for some scenarios. Loosely, for each label set (called a stream) that is pushed to a distributor, it will hash the labels and use the resulting value to look up replication_factor ingesters in the ring (which is a subcomponent that exposes a distributed hash table). It will then try to write the same data to all of them. This will generate an error if less than a quorum of writes succeeds. A quorum is defined as floor( replication_factor / 2 ) + 1. So, for our replication_factor of 3, we require that two writes succeed. If less than two writes succeed, the distributor returns an error and the write operation will be retried.

Caution
If a write is acknowledged by 2 out of 3 ingesters, we can tolerate the loss of one ingester but not two, as this would result in data loss.

The replication factor is not the only thing that prevents data loss, though, and its main purpose is to allow writes to continue uninterrupted during rollouts and restarts. The ingester component now includes a write ahead log (WAL) which persists incoming writes to disk to ensures they are not lost as long as the disk isn’t corrupted. The complementary nature of the replication factor and WAL ensures data isn’t lost unless there are significant failures in both mechanisms (that is, multiple ingesters die and lose/corrupt their disks).

Hashing

Distributors use consistent hashing in conjunction with a configurable replication factor to determine which instances of the ingester service should receive a given stream.

A stream is a set of logs associated to a tenant and a unique label set. The stream is hashed using both the tenant ID and the label set and then the hash is used to find the ingesters to send the stream to.

A hash ring, maintained by peer-to-peer communication using the Memberlist protocol, or stored in a Key-Value store such as Consul is used to achieve consistent hashing; all ingesters register themselves into the hash ring with a set of tokens they own. Each token is a random unsigned 32-bit number. Along with a set of tokens, ingesters register their state into the hash ring. The state JOINING, and ACTIVE may all receive write requests, while ACTIVE and LEAVING ingesters may receive read requests. When doing a hash lookup, distributors only use tokens for ingesters who are in the appropriate state for the request.

To do the hash lookup, distributors find the smallest appropriate token whose value is larger than the hash of the stream. When the replication factor is larger than 1, the next subsequent tokens (clockwise in the ring) that belong to different ingesters will also be included in the result.

The effect of this hash setup is that each token that an ingester owns is responsible for a range of hashes. If there are three tokens with values 0, 25, and 50, then a hash of 3 would be given to the ingester that owns the token 25; the ingester owning token 25 is responsible for the hash range of 1-25.

Quorum consistency

Since all distributors share access to the same hash ring, write requests can be sent to any distributor.

To ensure consistent query results, Loki uses Dynamo-style quorum consistency on reads and writes. This means that the distributor will wait for a positive response of at least one half plus one of the ingesters to send the sample to before responding to the client that initiated the send.

Ingester

The ingester service is responsible for persisting data and shipping it to long-term storage (Amazon Simple Storage Service, Google Cloud Storage, Azure Blob Storage, etc.) on the write path, and returning recently ingested, in-memory log data for queries on the read path.

Ingesters contain a lifecycler which manages the lifecycle of an ingester in the hash ring. Each ingester has a state of either PENDING, JOINING, ACTIVE, LEAVING, or UNHEALTHY:

PENDING is an Ingester’s state when it is waiting for a handoff from another ingester that is LEAVING. This only applies for legacy deployment modes.

Note
Handoff is a deprecated behavior mainly used in stateless deployments of ingesters, which is discouraged. Instead, it’s recommended using a stateful deployment model together with the write ahead log.
JOINING is an Ingester’s state when it is currently inserting its tokens into the ring and initializing itself. It may receive write requests for tokens it owns.
ACTIVE is an Ingester’s state when it is fully initialized. It may receive both write and read requests for tokens it owns.
LEAVING is an Ingester’s state when it is shutting down. It may receive read requests for data it still has in memory.
UNHEALTHY is an Ingester’s state when it has failed to heartbeat. UNHEALTHY is set by the distributor when it periodically checks the ring.

Each log stream that an ingester receives is built up into a set of many “chunks” in memory and flushed to the backing storage backend at a configurable interval.

Chunks are compressed and marked as read-only when:

The current chunk has reached capacity (a configurable value).
Too much time has passed without the current chunk being updated
A flush occurs.

Whenever a chunk is compressed and marked as read-only, a writable chunk takes its place.

If an ingester process crashes or exits abruptly, all the data that has not yet been flushed will be lost. Loki is usually configured to replicate multiple replicas (usually 3) of each log to mitigate this risk.

When a flush occurs to a persistent storage provider, the chunk is hashed based on its tenant, labels, and contents. This means that multiple ingesters with the same copy of data will not write the same data to the backing store twice, but if any write failed to one of the replicas, multiple differing chunk objects will be created in the backing store. See Querier for how data is deduplicated.

Timestamp Ordering

Loki is configured to accept out-of-order writes by default.

When not configured to accept out-of-order writes, the ingester validates that ingested log lines are in order. When an ingester receives a log line that doesn’t follow the expected order, the line is rejected and an error is returned to the user.

The ingester validates that log lines are received in timestamp-ascending order. Each log has a timestamp that occurs at a later time than the log before it. When the ingester receives a log that does not follow this order, the log line is rejected and an error is returned.

Logs from each unique set of labels are built up into “chunks” in memory and then flushed to the backing storage backend.

If an ingester process crashes or exits abruptly, all the data that has not yet been flushed could be lost. Loki is usually configured with a Write Ahead Log which can be replayed on restart as well as with a replication_factor (usually 3) of each log to mitigate this risk.

When not configured to accept out-of-order writes, all lines pushed to Loki for a given stream (unique combination of labels) must have a newer timestamp than the line received before it. There are, however, two cases for handling logs for the same stream with identical nanosecond timestamps:

If the incoming line exactly matches the previously received line (matching both the previous timestamp and log text), the incoming line will be treated as an exact duplicate and ignored.
If the incoming line has the same timestamp as the previous line but different content, the log line is accepted. This means it is possible to have two different log lines for the same timestamp.

Handoff

Warning
Handoff is deprecated behavior mainly used in stateless deployments of ingesters, which is discouraged. Instead, it’s recommended using a stateful deployment model together with the write ahead log.

By default, when an ingester is shutting down and tries to leave the hash ring, it will wait to see if a new ingester tries to enter before flushing and will try to initiate a handoff. The handoff will transfer all of the tokens and in-memory chunks owned by the leaving ingester to the new ingester.

Before joining the hash ring, ingesters will wait in PENDING state for a handoff to occur. After a configurable timeout, ingesters in the PENDING state that have not received a transfer will join the ring normally, inserting a new set of tokens.

This process is used to avoid flushing all chunks when shutting down, which is a slow process.

Filesystem support

While ingesters do support writing to the filesystem through BoltDB, this only works in single-process mode as queriers need access to the same back-end store and BoltDB only allows one process to have a lock on the DB at a given time.

Query frontend

The query frontend is an optional service providing the querier’s API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers. The querier service will be still required within the cluster, in order to execute the actual queries.

The query frontend internally performs some query adjustments and holds queries in an internal queue. In this setup, queriers act as workers which pull jobs from the queue, execute them, and return them to the query frontend for aggregation. Queriers need to be configured with the query frontend address (via the -querier.frontend-address CLI flag) in order to allow them to connect to the query frontends.

Query frontends are stateless. However, due to how the internal queue works, it’s recommended to run a few query frontend replicas to reap the benefit of fair scheduling. Two replicas should suffice in most cases.

Queueing

If no separate query scheduler component is used, the query frontend will also perform basic query queueing.

Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the total cost of ownership (TCO).
Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO).
Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants.

Splitting

The query frontend splits larger queries into multiple smaller queries, executing these queries in parallel on downstream queriers and stitching the results back together again. This prevents large (multi-day, etc) queries from causing out of memory issues in a single querier and helps to execute them faster.

Caching

Metric queries

The query frontend supports caching metric query results and reuses them on subsequent queries. If the cached results are incomplete, the query frontend calculates the required sub-queries and executes them in parallel on downstream queriers. The query frontend can optionally align queries with their step parameter to improve the cacheability of the query results. The result cache is compatible with any Loki caching backend (currently Memcached, Redis, and an in-memory cache).

Log queries

The query frontend also supports caching of log queries in form of a negative cache. This means that instead of caching the log results for quantized time ranges, Loki only caches empty results for quantized time ranges. This is more efficient than caching actual results because log queries are limited (usually 1000 results) and if you have a query over a long time range that matches only a few lines, and you only cache actual results, you’d still need to process a lot of data in addition to the data from the results cache in order to verify that nothing else matches.

Index stats queries

The query frontend caches index stats query results similar to the metric query results. This cache is only applicable when using single store TSDB.

Log volume queries

The query frontend caches log volume query results similar to the metric query results. This cache is only applicable when using single store TSDB.

Query scheduler

The query scheduler is an optional service providing more advanced queuing functionality than the query frontend. When using this component in the Loki deployment, query frontend pushes split up queries to the query scheduler which enqueues them in an internal in-memory queue. There is a queue for each tenant to guarantee the query fairness across all tenants. The queriers that connect to the query scheduler act as workers that pull their jobs from the queue, execute them, and return them to the query frontend for aggregation. Queriers therefore need to be configured with the query scheduler address (via the -querier.scheduler-address CLI flag) in order to allow them to connect to the query scheduler.

Query schedulers are stateless. However, due to the in-memory queue, it’s recommended to run more than one replica to keep the benefit of high availability. Two replicas should suffice in most cases.

Querier

The querier service is responsible for executing Log Query Language (LogQL) queries. The querier can handle HTTP requests from the client directly (in “single binary” mode, or as part of the read path in “simple scalable deployment”) or pull subqueries from the query frontend or query scheduler (in “microservice” mode).

It fetches log data from both the ingesters and from long-term storage. Queriers query all ingesters for in-memory data before falling back to running the same query against the backend store. Because of the replication factor, it is possible that the querier may receive duplicate data. To resolve this, the querier internally deduplicates data that has the same nanosecond timestamp, label set, and log message.

Index Gateway

The index gateway service is responsible for handling and serving metadata queries. Metadata queries are queries that look up data from the index. The index gateway is only used by “shipper stores”, such as single store TSDB or single store BoltDB.

The query frontend queries the index gateway for the log volume of queries so it can make a decision on how to shard the queries. The queriers query the index gateway for chunk references for a given query so they know which chunks to fetch and query.

The index gateway can run in simple or ring mode. In simple mode, each index gateway instance serves all indexes from all tenants. In ring mode, index gateways use a consistent hash ring to distribute and shard the indexes per tenant amongst available instances.

Compactor

The compactor service is used by “shipper stores”, such as single store TSDB or single store BoltDB, to compact the multiple index files produced by the ingesters and shipped to object storage into single index files per day and tenant. This makes index lookups more efficient.

To do so, the compactor downloads the files from object storage in a regular interval, merges them into a single one, uploads the newly created index, and cleans up the old files.

Additionally, the compactor is also responsible for log retention and log deletion.

In a Loki deployment, the compactor service is usually run as a single instance.

Ruler

The ruler service manages and evaluates rule and/or alert expressions provided in a rule configuration. The rule configuration is stored in object storage (or alternatively on local file system) and can be managed via the ruler API or directly by uploading the files to object storage.

Alternatively, the ruler can also delegate rule evaluation to the query frontend. This mode is called remote rule evaluation and is used to gain the advantages of query splitting, query sharding, and caching from the query frontend.

When running multiple rulers, they use a consistent hash ring to distribute rule groups amongst available ruler instances.

Pattern ingester

The optional pattern ingester component receives log data from the ingesters and scans the logs to detect and aggregate patterns. This can be useful for understanding the structure of your logs at scale. The pattern ingester is used by the pattern feature in Logs Drilldown, which lets you detect similar log lines and add or exclude them from your search.

The ingester uses a drain algorithm to identify related logs that share the same pattern, and maintain their counts over time. Patterns consist of a number, a string, and a Loki series identifier.

The pattern ingester exposes a query API, so you can fetch detected patterns. This API is used by the Patterns tab in the Grafana Logs Drilldown plugin.

This component is disabled by default and must be enabled in your Loki config file.

Bloom Planner

Warning
This feature is an experimental feature. Engineering and on-call support is not available. No SLA is provided.

The Bloom Planner service is responsible for planning the tasks for blooms creation. It runs as a singleton and provides a queue from which tasks are pulled by the Bloom Builders. The planning runs periodically and takes into account what blooms have already been built for a given day and tenant and what series need to be newly added.

This service is also used to apply blooms retention.

Bloom Builder

Warning
This feature is an experimental feature. Engineering and on-call support is not available. No SLA is provided.

The Bloom Builder service is responsible for processing the tasks created by the Bloom Planner. The Bloom Builder creates bloom blocks from structured metadata of log entries. The resulting blooms are grouped in bloom blocks spanning multiple series and chunks from a given day. This component also builds metadata files to track which blocks are available for each series and TSDB index file.

The service is stateless and horizontally scalable.

Bloom Gateway

Warning
This feature is an experimental feature. Engineering and on-call support is not available. No SLA is provided.

The Bloom Gateway service is responsible for handling and serving chunks filtering requests. The index gateway queries the Bloom Gateway when computing chunk references, or when computing shards for a given query. The gateway service takes a list of chunks and a filtering expression and matches them against the blooms, filtering out any chunks that do not match the given label filter expression.

The service is horizontally scalable. When running multiple instances, the client (Index Gateway) shards requests across instances based on the hash of the bloom blocks that are referenced.

Loki deployment modes

Thu, 09 Apr 2026 02:28:18 +0000

Loki deployment modes

Loki is a distributed system consisting of many microservices. It also has a unique build model where all of those microservices exist within the same binary.

You can configure the behavior of the single binary with the -target command-line flag to specify which microservices will run on startup. You can further configure each of the components in the loki.yaml file.

Because Loki decouples the data it stores from the software which ingests and queries it, you can easily redeploy a cluster under a different mode as your needs change, with minimal or no configuration changes.

Monolithic mode

The simplest mode of operation is the monolithic deployment mode. You enable monolithic mode by setting the -target=all command line parameter. This mode runs all of Loki’s microservice components inside a single process as a single binary or Docker image.

Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day.

You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the ring section of the loki.yaml file to share state between all instances, but the recommendation is to use microservices deployment mode if you need to scale your deployment.

You can configure high availability by running two Loki instances using memberlist_config configuration and a shared object store and setting the replication_factor to 3. You route traffic to all the Loki instances in a round robin fashion.

Query parallelization is limited by the number of instances and the setting max_query_parallelism which is defined in the loki.yaml file.

Simple Scalable

Note
Simple Scalable Deployment (SSD) mode is being deprecated. The timeline for the deprecation is to be determined (TBD), but will happen before Loki 4.0 is released.

The simple scalable deployment is the default configuration installed by the Loki Helm Chart. This deployment mode is the easiest way to deploy Loki at scale. It strikes a balance between deploying in monolithic mode or deploying each component as a separate microservice. Simple scalable deployment is also referred to as SSD.

Note
This deployment mode is sometimes referred to by the acronym SSD for simple scalable deployment, not to be confused with solid state drives. Loki uses an object store.

Loki’s simple scalable deployment mode separates execution paths into read, write, and backend targets. These targets can be scaled independently, letting you customize your Loki deployment to meet your business needs for log ingestion and log query so that your infrastructure costs better match how you use Loki.

The simple scalable deployment mode can scale close to a TB of logs per day. Even though scaling it further may be possible, at that scale, the microservices mode will be a better choice in terms of scalability and ease of operations

The three execution paths in simple scalable mode are each activated by appending the following arguments to Loki on startup:

-target=write - The write target is stateful and is controlled by a Kubernetes StatefulSet. It contains the following components:
- Distributor
- Ingester
-target=read - The read target is stateless and can be run as a Kubernetes Deployment that can be scaled automatically (Note that in the official helm chart it is currently deployed as a stateful set). It contains the following components:
- Query Frontend
- Querier
-target=backend - The backend target is stateful, and is controlled by a Kubernetes StatefulSet. Contains the following components:
- Compactor
- Index Gateway
- Query Scheduler
- Ruler
- Bloom Planner (experimental)
- Bloom Builder (experimental)
- Bloom Gateway (experimental)

The simple scalable deployment mode requires a reverse proxy to be deployed in front of Loki, to direct client API requests to either the read or write nodes. The Loki Helm chart includes a default reverse proxy configuration, using Nginx.

Microservices mode

The microservices deployment mode runs components of Loki as distinct processes. The microservices deployment is also referred to as a Distributed deployment. Each process is invoked specifying its target. For release 3.3 the components are:

Bloom Builder (experimental)
Bloom Gateway (experimental)
Bloom Planner (experimental)
Compactor
Distributor
Index Gateway
Ingester
Overrides Exporter
Querier
Query Frontend
Query Scheduler
Ruler
Table Manager (deprecated)

Tip
You can see the complete list of targets for your version of Loki by running Loki with the flag -list-targets, for example:
Bash
docker run docker.io/grafana/loki:3.7.0 -config.file=/etc/loki/local-config.yaml -list-targets

Running components as individual microservices provides more granularity, letting you scale each component as individual microservices, to better match your specific use case.

Microservices mode deployments can be more efficient Loki installations. However, they are also the most complex to set up and maintain.

Microservices mode is only recommended for very large Loki clusters or for operators who need more precise control over scaling and cluster operations.

Microservices mode is designed for Kubernetes deployments. A community-supported Helm chart is available for deploying Loki in microservices mode.

Understand labels

Thu, 09 Apr 2026 02:28:18 +0000

Understand labels

Labels are a crucial part of Loki. They allow Loki to organize and group together log messages into log streams. Each log stream must have at least one label to be stored and queried in Loki.

In this topic we’ll learn about labels and why your choice of labels is important when shipping logs to Loki.

Note
Labels are intended to store low-cardinality values that describe the source of your logs. If you frequently search high-cardinality data in your logs, you should use structured metadata.

Understand labels

In Loki, the content of each log line is not indexed. Instead, log entries are grouped into streams which are indexed with labels.

A label is a key-value pair, for example all of the following are labels:

deployment_environment = development
cloud_region = us-west-1
namespace = grafana-server

A set of log messages which shares all the labels above would be called a log stream. When Loki performs searches, it first looks for all messages in your chosen stream, and then iterates through the logs in the stream to perform your query.

Labeling will affect your queries, which in turn will affect your dashboards. It’s worth spending the time to think about your labeling strategy before you begin ingesting logs to Loki.

Default labels for all users

Loki does not parse or process your log messages on ingestion. However, depending on which client you use to collect logs, you may have some labels automatically applied to your logs.

service_name

Loki automatically tries to populate a default service_name label while ingesting logs. The service name label is used to find and explore logs in the following Grafana and Grafana Cloud features:

Logs Drilldown
Grafana Cloud Application Observability

Note
If you are already applying a service_name, Loki will use that value. For example, if you are using the Kubernetes monitoring Helm Chart, the Alloy configuration applies a service_name by default.

Loki will attempt to create the service_name label by looking for the following labels in this order:

service_name
service
app
application
name
app_kubernetes_io_name
container
container_name
component
workload
job

If no label is found matching the list, a value of unknown_service is applied.

You can change this list by providing a list of labels to discover_service_name in the limits_config block. If you are using Grafana Cloud, contact support to configure this setting.

Default labels for OpenTelemetry

If you are using either Grafana Alloy or the OpenTelemetry Collector as your Loki client, then Loki automatically assigns some of the OTel resource attributes as labels. Resource attributes map well to index labels in Loki, since both usually identify the source of the logs.

By default, the following resource attributes will be stored as labels, with periods (.) replaced with underscores (_), while the remaining attributes are stored as structured metadata with each log entry:

cloud.availability_zone
cloud.region
container.name
deployment.environment.name
k8s.cluster.name
k8s.container.name
k8s.cronjob.name
k8s.daemonset.name
k8s.deployment.name
k8s.job.name
k8s.namespace.name
k8s.pod.name
k8s.replicaset.name
k8s.statefulset.name
service.instance.id
service.name
service.namespace

Note
Because Loki has a default limit of 15 index labels, we recommend storing only select resource attributes as labels. Although the default config selects more than 15 Resource Attributes, some are mutually exclusive.

Tip
For Grafana Cloud Logs, see the current OpenTelemetry guidance. The Faro specific attributes app_id, kind, and app_key are promoted to labels for Grafana Cloud Logs but not Loki.

The default list of resource attributes to store as labels can be configured using default_resource_attributes_as_index_labels under the distributor’s otlp_config. You can set global limits using limits_config.otlp_config. If you are using Grafana Cloud, contact support to configure this setting.

Caution
Because of the potential for high cardinality, k8s.pod.name and service.instance.id are no longer recommended as default labels. But because removing these resource attributes from the default labels would be a breaking change for existing users, they have not yet been deprecated as default labels. If you are a new user of Grafana Loki, we recommend that you modify your Alloy or OpenTelemetry Collector configuration to convert k8s.pod.name and service.instance.id from index labels to structured metadata. For sample configurations, refer to Modify default labels.

Labeling is iterative

You want to start with a small set of labels. While accepting the default labels assigned by Grafana Alloy or the OpenTelemetry Collector or the Kubernetes Monitoring Helm chart may meet your needs, over time you may find that you need to modify your labeling strategy.

Once you understand how your first set of labels works and you understand how to apply and query with those labels, you may find that they don’t meet your query patterns. You may need to modify or change your labels and test your queries again.

Settling on the right labels for your business needs may require multiple rounds of testing. This should be expected as you continue to tune your Loki environment to meet your business requirements.

Create low cardinality labels

Cardinality refers to the combination of unique labels and values which impacts the number of log streams you create. High cardinality causes Loki to build a huge index and to flush thousands of tiny chunks to the object store. Loki performs very poorly when your labels have high cardinality. If not accounted for, high cardinality will significantly reduce the performance and cost-effectiveness of Loki.

High cardinality can result from using labels with an unbounded or large set of possible values, such as timestamp or ip_address or applying too many labels, even if they have a small and finite set of values.

High cardinality can lead to significant performance degradation. Prefer fewer labels, which have bounded values.

Creating custom labels

Tip
Many log collectors such as Grafana Alloy, or the Kubernetes Monitoring Helm chart, will automatically assign appropriate labels for you, so you don’t need to create your own labeling strategy. For most use cases, you can just accept the default labels.

Usually, labels describe the source of the log, for example:

the namespace or additional logical grouping of applications
cluster, and/or region of where the logs were produced
the filename of the source log file on disk
the hostname where the log was produced, if the environment has individually named machines or virtual machines. If you have an environment with ephemeral machines or virtual machines, the hostname should be stored in structured metadata.

If your logs had the example labels above, then you might query them in LogQL like this:

{namespace="mynamespace", cluster="cluster123" filename="/var/log/myapp.log"}

Unlike index-based log aggregators, Loki doesn’t require you to create a label for every field that you might wish to search in your log content. Labels are only needed to organize and identify your log streams. Loki performs searches by iterating over a log stream in a highly parallelized fashion to look for a given string.

For more information on how Loki performs searches, see the Query section.

This means that you don’t need to add labels for things inside the log message, such as:

log level
log message
exception name

That being said, in some cases you may wish to add some extra labels, which can help to narrow down your log streams even further. When adding custom labels, follow these principles:

DO use fewer labels, aim to have 10 - 15 labels at a maximum. Fewer labels means a smaller index, which leads to better performance.
DO be as specific with your labels you can be, the less searching that Loki has to do, the faster your result is returned.
DO create labels with long-lived values, not unbounded values. To be a good label, we want something that has a stable set of values over time – even if there are a lot of them. If just one label value changes, this creates a new stream.
DO create labels based on terms that your users will actually be querying on.
DON’T create labels for very specific searches (for example, user ID or customer ID) or seldom used searches (searches performed maybe once a year).

Label format

Loki places the same restrictions on label naming as Prometheus:

It may contain ASCII letters and digits, as well as underscores and colons. It must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*.
Unsupported characters in the label should be converted to an underscore. For example, the label app.kubernetes.io/name should be written as app_kubernetes_io_name.
However, do not begin and end your label names with double underscores, as this naming convention is used for internal labels, for example, _stream_shard_, that are hidden by default in the label browser, query builder, and autocomplete to avoid creating confusion for users.

In Loki, you do not need to add labels based on the content of the log message.

Labels and ingestion order

Loki supports ingesting out-of-order log entries. Out-of-order writes are enabled globally by default, but can be disabled/enabled on a cluster or per-tenant basis. If you plan to ingest out-of-order log entries, your label selection is important. We recommend trying to find a way to use labels to separate the streams so they can be ingested separately.

Entries in a given log stream (identified by a given set of label names & values) must be ingested in order, within the default two hour time window. If you try to send entries that are too old for a given log stream, Loki will respond with the error too far behind.

For systems with different ingestion delays and shipping, use labels to create separate streams. Instead of:

{environment="production"}

You may separate the log stream into:

{environment="production", app="slow_app"} {environment="production", app="fast_app"}

Now the “fast_app” and “slow_app” will ship logs to different streams, allowing each to maintain their order of ingestion.

Loki labels examples

The way that labels are added to logs is configured in the client that you use to send logs to Loki. The specific configuration will be different for each client.

Alloy example

Grafana Labs recommends using Grafana Alloy to send logs to Loki. Here is an example configuration:


local.file_match "tmplogs" {
    path_targets = [{"__path__" = "/tmp/alloy-logs/*.log"}]
}

loki.source.file "local_files" {
    targets    = local.file_match.tmplogs.targets
    forward_to = [loki.process.add_new_label.receiver]
}

loki.process "add_new_label" {
    // Extract the value of "level" from the log line and add it to the extracted map as "extracted_level"
    // You could also use "level" = "", which would extract the value of "level" and add it to the extracted map as "level"
    // but to make it explicit for this example, we will use a different name.
    //
    // The extracted map will be covered in more detail in the next section.
    stage.logfmt {
        mapping = {
            "extracted_level" = "level",
        }
    }

    // Add the value of "extracted_level" from the extracted map as a "level" label
    stage.labels {
        values = {
            "level" = "extracted_level",
        }
    }

    forward_to = [loki.relabel.add_static_label.receiver]
}

loki.relabel "add_static_label" {
    forward_to = [loki.write.local_loki.receiver]

    rule {
        target_label = "os"
        replacement  = constants.os
    }
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}

Cardinality examples

The two previous examples use statically defined labels with a single value; however, there are ways to dynamically define labels. Let’s take a look using the Apache log and a massive regex you could use to parse such a log line:

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

- job_name: system
  pipeline_stages:
    - regex:
        expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status_code>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"]*)\"?\\s?\"?(?P<useragent>[^\"]*)?\"?$"
    - labels:
        action:
        status_code:
  static_configs:
    - targets:
        - localhost
      labels:
        job: apache
        env: dev
        __path__: /var/log/apache.log

This regex matches every component of the log line and extracts the value of each component into a capture group. Inside the pipeline code, this data is placed in a temporary data structure that allows use for several purposes during the processing of that log line (at which point that temp data is discarded). Much more detail about this can be found in the Alloy loki.process documentation.

From that regex, we will be using two of the capture groups to dynamically set two labels based on content from the log line itself:

action (for example, action="GET", action="POST")

status_code (for example, status_code="200", status_code="400")

And now let’s walk through a few example lines:

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

In Loki the following streams would be created:

{job="apache",env="dev",action="GET",status_code="200"} 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="POST",status_code="200"} 11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="GET",status_code="400"} 11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
{job="apache",env="dev",action="POST",status_code="400"} 11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

Those four log lines would become four separate streams and start filling four separate chunks.

Any additional log lines that match those combinations of labels/values would be added to the existing stream. If another unique combination of labels comes in (for example, status_code="500") another new stream is created.

Imagine now if you set a label for ip. Not only does every request from a user become a unique stream. Every request with a different action or status_code from the same user will get its own stream.

Doing some quick math, if there are maybe four common actions (GET, PUT, POST, DELETE) and maybe four common status codes (although there could be more than four!), this would be 16 streams and 16 separate chunks. Now multiply this by every user if we use a label for ip. You can quickly have thousands or tens of thousands of streams.

Consistent hash rings

Thu, 09 Apr 2026 02:28:18 +0000

Consistent hash rings

Consistent hash rings are incorporated into Loki cluster architectures to

aid in the sharding of log lines
implement high availability
ease the horizontal scale up and scale down of clusters. There is less of a performance hit for operations that must rebalance data.

Hash rings connect instances of a single type of component when

there are a set of Loki instances in monolithic deployment mode
there are multiple read components or multiple write components in simple scalable deployment mode
there are multiple instances of one type of component in microservices mode

Not all Loki components are connected by hash rings. These components need to be connected into a hash ring:

distributors
ingesters
query schedulers
compactors
rulers

These components can optionally be connected into a hash ring:

index gateway

In an architecture that has three distributors and three ingesters defined, the hash rings for these components connect the instances of same-type components.

Each node in the ring represents an instance of a component. Each node has a key-value store that holds communication information for each of the nodes in that ring. Nodes update the key-value store periodically to keep the contents consistent across all nodes. For each node, the key-value store holds:

an ID of the component node
component address, used by other nodes as a communication channel
an indication of the component node’s health

Configuring rings

Define ring configuration within the common.ring block.

Use the memberlist key-value store type unless there is a compelling reason to use a different key-value store type. memberlist uses a gossip protocol to propagate information to all the nodes to guarantee the eventual consistency of the key-value store contents.

There are additional configuration options for distributor rings, ingester rings, and ruler rings. These options are for advanced, specialized use only. These options are defined within the distributor.ring block for distributors, the ingester.lifecycler.ring block for ingesters, and the ruler.ring block for rulers.

About the distributor ring

Distributors use the information in their key-value store to keep a count of the quantity of distributors in the distributor ring. The count further informs cluster limits.

About the ingester ring

Ingester ring information in the key-value stores is used by distributors. The information lets the distributors shard log lines, determining which ingester or set of ingesters a distributor sends log lines to.

About the query scheduler ring

Query schedulers use the information in their key-value store for service discovery of the schedulers. This allows queriers to connect to all available schedulers, and it allows schedulers to connect to all available query frontends, effectively creating a single queue that aids in balancing the query load.

About the compactor ring

Compactors use the information in the key-value store to identify a single compactor instance that will be responsible for compaction. The compactor is only enabled on the responsible instance, despite the compactor target being on multiple instances.

About the ruler ring

The ruler ring is used to determine which rulers evaluate which rule groups.

About the index gateway ring

The index gateway ring is used to determine which gateway is responsible for which tenant’s indexes when queried by rulers or queriers.