Manage and debug errors on Grafana Labs

Troubleshoot Loki operations

Thu, 09 Apr 2026 00:47:14 +0000

Troubleshoot Loki operations

This guide helps you troubleshoot errors that occur during Loki operations, including configuration issues, storage backend problems, cluster communication failures, and service component errors. These errors are distinct from ingestion (write path) and query (read path) errors covered in separate troubleshooting topics.

Before you begin, ensure you have the following:

Access to Loki logs and metrics
Permissions to view and modify Loki configuration
Understanding of your deployment topology (single binary/monolithic, simple scalable, microservices/distributed)

Configuration errors

Configuration errors occur during Loki startup or when loading runtime configuration. These errors prevent Loki from starting or operating correctly.

Error: Multiple config errors found

Error message:

MULTIPLE CONFIG ERRORS FOUND, PLEASE READ CAREFULLY
<list of configuration errors>

Cause:

Multiple configuration validation errors were detected during startup. Loki aggregates all configuration errors rather than failing on the first one.

Resolution:

Review all listed errors carefully - each error message describes a specific configuration problem.
Check your configuration file for syntax errors and invalid values.

Validate your configuration before applying:

loki -config.file=/path/to/config.yaml -verify-config

Properties:

Enforced by: Loki startup
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Too many storage configs

Error message:

too many storage configs provided in the common config, please only define one storage backend

Cause:

Multiple storage backends are configured in the common configuration section. Loki requires a single storage backend for the common config.

Resolution:

Use only one storage backend in your common config:

common:
  storage:
    # Choose only ONE of the following:
    s3:
      endpoint: s3.amazonaws.com
      bucketnames: loki-data
    # OR
    gcs:
      bucket_name: loki-data
    # OR
    azure:
      container_name: loki-data

For multiple storage backends, configure them explicitly in specific sections rather than common config.

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Persist tokens path prefix required

Error message:

if persist_tokens is true, path_prefix MUST be defined

Cause:

The persist_tokens option is enabled for a ring but no path_prefix is specified. Loki needs a path to store the token file.

Resolution:

Set the path prefix:

common:
  path_prefix: /var/loki
  persist_tokens: true

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
    tokens_file_path: /var/loki/tokens

Or disable persist_tokens if you don’t need token persistence:
YAML
```
common:
  persist_tokens: false
```

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Conflicting gRPC client configs

Error message:

both `grpc_client_config` and (`query_frontend_grpc_client` or `query_scheduler_grpc_client`) are set at the same time. Please use only `query_frontend_grpc_client` and `query_scheduler_grpc_client`

Cause:

Both the deprecated grpc_client_config and the newer specific gRPC client configs are set. These are mutually exclusive.

Resolution:

Remove the deprecated config and use specific gRPC client configs:

# Remove this:
# grpc_client_config: ...

# Use these instead:
query_frontend_grpc_client:
  max_recv_msg_size: 104857600
query_scheduler_grpc_client:
  max_recv_msg_size: 104857600

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Schema v13 required for structured metadata

Error message:

CONFIG ERROR: schema v13 is required to store Structured Metadata and use native OTLP ingestion, your schema version is <version>. Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update to schema v13 or newer before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure

Cause:

Structured metadata is enabled but the active schema version is older than v13. Structured metadata requires schema v13 or newer.

Resolution:

Disable structured metadata temporarily:

limits_config:
  allow_structured_metadata: false

Update your schema config to v13 or newer:

schema_config:
  configs:
    - from: "2024-04-01"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Re-enable structured metadata after the schema migration is complete.

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: TSDB index type required for structured metadata

Error message:

CONFIG ERROR: `tsdb` index type is required to store Structured Metadata and use native OTLP ingestion, your index type is `<type>` (defined in the `store` parameter of the schema_config). Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update the schema to use index type `tsdb` before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure

Cause:

Structured metadata is enabled but the active index type is not TSDB. Structured metadata requires the TSDB index type.

Resolution:

Disable structured metadata temporarily and migrate to the TSDB index type:

limits_config:
  allow_structured_metadata: false

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: s3
      schema: v13

Re-enable structured metadata after migrating to TSDB.

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: TSDB directories not configured

Error message:

CONFIG ERROR: `tsdb` index type is configured in at least one schema period, however, `storage_config`, `tsdb_shipper`, `active_index_directory` is not set, please set this directly or set `path_prefix:` in the `common:` section

Or:

CONFIG ERROR: `tsdb` index type is configured in at least one schema period, however, `storage_config`, `tsdb_shipper`, `cache_location` is not set, please set this directly or set `path_prefix:` in the `common:` section

Cause:

The TSDB index type is configured in the schema but required local directories for index files are not set.

Resolution:

Set the common path prefix (simplest approach):
YAML
```
common:
  path_prefix: /var/loki
```

Or configure directories explicitly:

storage_config:
  tsdb_shipper:
    active_index_directory: /var/loki/tsdb-index
    cache_location: /var/loki/tsdb-cache

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Compactor working directory empty

Error message:

CONFIG ERROR: `compactor:` `working_directory:` is empty, please set a valid directory or set `path_prefix:` in the `common:` section

Cause:

The compactor requires a working directory for index compaction, but none is configured.

Resolution:

Set the common path prefix:
YAML
```
common:
  path_prefix: /var/loki
```
Or set the compactor working directory explicitly:
YAML
```
compactor:
  working_directory: /var/loki/compactor
```

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Index cache validity conflict

Error message:

CONFIG ERROR: the active index is <type> which is configured to use an `index_cache_validity` (TTL) of <duration>, however the chunk_retain_period is <duration> which is LESS than the `index_cache_validity`. This can lead to query gaps, please configure the `chunk_retain_period` to be greater than the `index_cache_validity`

Cause:

The chunk retain period is shorter than the index cache validity (TTL), which can cause query gaps where data exists in the index cache but the chunks have already been flushed and removed from ingesters.

Resolution:

Increase the chunk retain period to be greater than the index cache validity:

ingester:
  chunk_retain_period: 15m  # Must be > index_cache_validity

storage_config:
  index_cache_validity: 5m

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Invalid target with legacy read mode

Error message:

CONFIG ERROR: invalid target, cannot run backend target with legacy read mode

Cause:

The backend target is configured while legacy read mode is enabled. These are incompatible deployment configurations.

Resolution:

Disable legacy read mode if using the backend target:
YAML
```
# Remove or set to false:
legacy_read_mode: false
```
Or use a different target compatible with legacy read mode.

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Unrecognized index or store type

Error message:

unrecognized `store` (index) type `<type>`, choose one of: <supported_types>

Or:

unrecognized `object_store` type `<type>`, which also does not match any named_stores. Choose one of: <supported_types>. Or choose a named_store

Cause:

The schema configuration references an index type or object store type that Loki does not recognize.

Resolution:

Use a supported index type: tsdb (recommended) or boltdb-shipper
Use a supported object store type: s3, gcs, azure, swift, filesystem, bos

Or reference a valid named store defined in your configuration:

storage_config:
  named_stores:
    aws:
      my-store:
        endpoint: s3.amazonaws.com
        bucketnames: my-bucket

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: my-store  # References the named store

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Overrides exporter requires runtime configuration

Error message:

overrides-exporter has been enabled, but no runtime configuration file was configured

Cause:

The overrides-exporter target is enabled but no runtime configuration file is provided. The overrides-exporter needs a runtime config to expose tenant-specific limit overrides as metrics.

Resolution:

Configure a runtime configuration file:

runtime_config:
  file: /etc/loki/runtime-config.yaml

Or disable the overrides-exporter if not needed by removing it from your target list.

Properties:

Enforced by: Module initialization
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Invalid override for tenant

Error message:

invalid override for tenant <tenant>: <details>

Cause:

The runtime configuration file contains an invalid override for a specific tenant. The override failed validation.

Resolution:

Review the runtime config file for the specified tenant.
Validate the override values against the limits configuration schema.
Fix invalid values such as negative durations, invalid label matchers, or out-of-range settings.

Properties:

Enforced by: Runtime configuration loader
Retryable: No (runtime config must be fixed)
HTTP status: N/A (runtime config reload failure)
Configurable per tenant: Yes

Error: Retention period too short

Error message:

retention period must be >= 24h was <duration>

Cause:

A stream-level retention rule specifies a retention period shorter than 24 hours, which is the minimum allowed.

Resolution:

Set retention periods to at least 24 hours:

limits_config:
  retention_stream:
    - selector: '{namespace="dev"}'
      priority: 1
      period: 24h  # Must be >= 24h

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: Yes

Error: Invalid query store max look back period

Error message:

it is an error to specify a non zero `query_store_max_look_back_period` value when using any object store other than `filesystem`

Cause:

The query_store_max_look_back_period is set to a non-zero value with a storage backend other than filesystem. This setting only applies to local filesystem storage.

Resolution:

Remove the setting if using object storage:

# Remove or set to 0:
query_store_max_look_back_period: 0

Or use filesystem storage if this setting is needed for local development.

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Authentication and tenant errors

Authentication and tenant errors occur when requests are missing required tenant identification or when tenant IDs are invalid. In multi-tenant mode, every request must include a valid tenant ID.

Error: No org ID

Error message:

no org id

Cause:

A request was made to Loki without the required X-Scope-OrgID header. In multi-tenant mode, every request must identify the tenant.

Resolution:

Add the X-Scope-OrgID header to your requests:

curl -H "X-Scope-OrgID: my-tenant" http://loki:3100/loki/api/v1/push ...

For Grafana, configure the tenant ID in the Loki data source settings under “HTTP Headers”.

For Alloy, set the tenant ID in the loki.write component:

loki.write "default" {
  endpoint {
    url       = "http://loki:3100/loki/api/v1/push"
    tenant_id = "my-tenant"
  }
}

Disable multi-tenancy for single-tenant deployments:
YAML
```
auth_enabled: false
```

Properties:

Enforced by: Authentication middleware
Retryable: Yes (with tenant ID)
HTTP status: 401 Unauthorized
Configurable per tenant: No

Error: Multiple org IDs present

Error message:

multiple org IDs present

Cause:

The request contains multiple different tenant IDs, but the operation requires a single tenant. This can happen when a request is forwarded through multiple proxies that each inject a tenant ID.

Resolution:

Ensure only one tenant ID is set in the X-Scope-OrgID header.
Check proxy configurations for conflicting tenant ID injection.
For cross-tenant queries, use pipe-separated tenant IDs only where supported:
Bash
```
curl -H "X-Scope-OrgID: tenant1|tenant2" http://loki:3100/loki/api/v1/query ...
```

Properties:

Enforced by: Tenant resolver
Retryable: Yes (with correct tenant ID)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Tenant ID too long

Error message:

tenant ID is too long: max 150 characters

Cause:

The tenant ID exceeds the maximum allowed length of 150 characters.

Resolution:

Use a shorter tenant ID (maximum 150 characters).

Properties:

Enforced by: Tenant validation
Retryable: Yes (with valid tenant ID)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Unsafe tenant ID

Error message:

tenant ID is '.' or '..'

Cause:

The tenant ID is set to . or .., which are reserved filesystem path components and could cause path traversal issues.

Resolution:

Choose a different tenant ID that is not . or ...

Properties:

Enforced by: Tenant validation
Retryable: Yes (with valid tenant ID)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Tenant ID contains unsupported character

Error message:

tenant ID '<id>' contains unsupported character '<char>'

Cause:

The tenant ID contains characters that are not allowed. Tenant IDs must consist of alphanumeric characters, hyphens, underscores, and periods.

Resolution:

Use only supported characters in your tenant ID: letters, numbers, hyphens (-), underscores (_), and periods (.).

Properties:

Enforced by: Tenant validation
Retryable: Yes (with valid tenant ID)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Deletion not available for tenant

Error message:

deletion is not available for this tenant

Cause:

A delete request was submitted for a tenant that does not have deletion enabled. Log deletion must be explicitly enabled per tenant.

Resolution:

Enable deletion for the tenant in the runtime configuration:
YAML
```
overrides:
  my-tenant:
    deletion_mode: filter-and-delete  # Or "filter-only"
```
Valid deletion modes:
- disabled - Deletion is not allowed (default)
- filter-only - Lines matching delete requests are filtered at query time but not physically deleted
- filter-and-delete - Lines are filtered at query time and physically deleted during compaction

Ensure the compactor is configured for retention:

compactor:
  retention_enabled: true
  delete_request_store: s3

Properties:

Enforced by: Compactor deletion handler
Retryable: No (configuration must change)
HTTP status: 403 Forbidden
Configurable per tenant: Yes

Storage backend errors

Storage backend errors occur when Loki cannot communicate with or properly configure object storage (Amazon S3, Google Cloud Services, Microsoft Azure, Swift, or filesystem).

Error: Unsupported storage backend

Error message:

unsupported storage backend

Cause:

The specified storage backend type is not recognized. This typically occurs when a typo exists in the storage type configuration.

Resolution:

Use a valid storage backend type:
- s3 - Amazon S3 or S3-compatible storage
- gcs - Google Cloud Storage
- azure - Azure Blob Storage
- swift - OpenStack Swift
- filesystem - Local filesystem
- bos - Baidu Object Storage
YAML
```
storage_config:
  boltdb_shipper:
    shared_store: s3  # Must be one of the valid types
```

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Invalid characters in storage prefix

Error message:

storage prefix contains invalid characters, it may only contain digits, English alphabet letters and dashes

Cause:

The storage path prefix contains invalid characters. Only alphanumeric characters and dashes are allowed.

Resolution:

Use valid characters in your storage prefix:

storage_config:
  # Invalid: prefix_with_underscore_or/special chars
  # Valid: my-loki-data or lokilogs123
  aws:
    s3: s3://my-bucket/my-loki-data

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Unsupported S3 SSE type

Error message:

unsupported S3 SSE type

Cause:

The S3 server-side encryption (SSE) type is not supported. Loki supports specific SSE types.

Resolution:

Use a supported SSE type:
YAML
```
storage_config:
  aws:
    sse:
      type: SSE-S3    # Or SSE-KMS
```
Supported types:
- SSE-S3 - Server-side encryption with Amazon S3-managed keys
- SSE-KMS - Server-side encryption with AWS KMS-managed keys

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Invalid S3 SSE encryption context

Error message:

invalid S3 SSE encryption context

Cause:

The SSE-KMS encryption context is malformed and cannot be parsed as valid JSON.

Resolution:

Provide valid JSON for the encryption context:

storage_config:
  aws:
    sse:
      type: SSE-KMS
      kms_key_id: alias/my-key
      kms_encryption_context: '{"key": "value"}'  # Valid JSON

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Invalid S3 endpoint prefix

Error message:

the endpoint must not prefixed with the bucket name

Cause:

The S3 endpoint incorrectly includes the bucket name as a prefix. This can cause path-style vs virtual-hosted-style URL issues.

Resolution:

Remove the bucket name from the endpoint and configure it separately:

storage_config:
  aws:
    # Incorrect:
    # endpoint: my-bucket.s3.amazonaws.com

    # Correct:
    endpoint: s3.amazonaws.com
    bucketnames: my-bucket

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Invalid STS endpoint

Error message:

sts-endpoint must be a valid url

Cause:

The AWS STS (Security Token Service) endpoint URL is malformed or invalid.

Resolution:

Provide a valid URL for the STS endpoint:

storage_config:
  aws:
    sts_endpoint: https://sts.us-east-1.amazonaws.com

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Azure connection string malformed

Error message:

connection string is either blank or malformed. The expected connection string should contain key value pairs separated by semicolons. For example 'DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>;EndpointSuffix=core.windows.net'

Cause:

The Azure storage connection string is missing or doesn’t follow the expected format.

Resolution:

Use a valid connection string format:

storage_config:
  azure:
    # Use account credentials:
    account_name: myaccount
    account_key: mykey

    # Or connection string:
    connection_string: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"

Verify the connection string in Azure Portal under Storage Account > Access Keys.

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Unrecognized named storage config

Error message:

unrecognized named storage config <name>

Or for specific backends:

unrecognized named s3 storage config <name>
unrecognized named gcs storage config <name>
unrecognized named azure storage config <name>
unrecognized named filesystem storage config <name>
unrecognized named swift storage config <name>

Or for an unrecognized store type:

unrecognized named storage type: <storeType>

Cause:

A named storage configuration referenced in the schema config doesn’t exist in the named stores configuration.

Resolution:

Define the named store in your configuration:

storage_config:
  named_stores:
    aws:
      my-s3-store:  # This name must match the reference
        endpoint: s3.amazonaws.com
        bucketnames: my-bucket

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: my-s3-store  # References the named store above

Check spelling of the store name in both the definition and reference.

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Cache errors

Cache errors occur when Loki cannot connect to or communicate with caching backends (Memcached, Redis).

Error: Redis client setup failed

Error message:

redis client setup failed: <details>

Cause:

Loki cannot establish a connection to the Redis server. Common causes include:

Incorrect Redis endpoint
Network connectivity issues
Authentication failures
TLS configuration problems

Resolution:

Verify Redis connectivity from the Loki host:
Bash
```
redis-cli -h <REDIS-HOST> -p <REDIS-PORT> ping
```

Check the Redis endpoint configuration:

chunk_store_config:
  chunk_cache_config:
    redis:
      endpoint: redis:6379
      timeout: 500ms

Configure authentication if required:

chunk_store_config:
  chunk_cache_config:
    redis:
      endpoint: redis:6379
      password: ${REDIS_PASSWORD}

Properties:

Enforced by: Cache client initialization
Retryable: Yes (with correct configuration)
HTTP status: N/A (startup failure or degraded operation)
Configurable per tenant: No

Error: Could not lookup Redis host

Error message:

could not lookup host: <hostname>

Cause:

DNS resolution failed for the Redis hostname.

Resolution:

Verify DNS resolution:
Bash
```
nslookup redis-host
```

Use an IP address if DNS is not available:

chunk_store_config:
  chunk_cache_config:
    redis:
      endpoint: 10.0.0.100:6379

Check your DNS configuration and network settings.

Properties:

Enforced by: DNS resolution
Retryable: Yes
HTTP status: N/A
Configurable per tenant: No

Error: Unexpected Redis PING response

Error message:

redis: Unexpected PING response "<response>"

Cause:

The Redis server returned an unexpected response to a PING command. This could indicate:

The endpoint is not a Redis server
A proxy or load balancer is interfering
Redis is in an error state

Resolution:

Verify the endpoint is actually a Redis server.
Check Redis health:
Bash
```
redis-cli -h <HOST> -p <PORT> INFO
```
Review proxy configurations if using a load balancer in front of Redis.

Properties:

Enforced by: Redis health check
Retryable: Yes
HTTP status: N/A
Configurable per tenant: No

Error: Multiple cache systems not supported

Error message:

use of multiple cache storage systems is not supported

Cause:

Both Memcached and Redis cache backends are configured for the same cache type. Only one caching backend is supported per cache type.

Resolution:

Choose one cache backend per cache type:

chunk_store_config:
  chunk_cache_config:
    # Use either memcached OR redis, not both
    redis:
      endpoint: redis:6379
    memcached: {}  # Remove this

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: No cache configured

Error message:

no cache configured

Cause:

A results cache is required for the query frontend but no cache configuration was provided.

Resolution:

Configure a cache backend:

query_range:
  results_cache:
    cache:
      memcached:
        expiration: 1h
      memcached_client:
        addresses: memcached:11211

Or disable results caching if not needed:
YAML
```
query_range:
  cache_results: false
```

Properties:

Enforced by: Query frontend initialization
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Error loading cache generation numbers

Error message:

msg="error loading cache generation numbers" err="unexpected status code: 403"

Or from the compactor HTTP client:

msg="error getting cache gen numbers from the store" err="unexpected status code: 403"

Metric: loki_delete_cache_gen_load_failures_total

Cause:

Loki uses cache generation numbers to invalidate query caches when log deletion requests are processed. The cache generation number loader periodically fetches these numbers from the compactor. When the compactor returns HTTP 403, it means the deletion API is not enabled for the tenant. The loader logs this error and increments the loki_delete_cache_gen_load_failures_total metric.

Other non- 403 causes include:

The compactor is unreachable (network or DNS issues)
The compactor is not running or not ready
TLS or authentication misconfiguration between the querier and compactor

Resolution:

Check if deletion is intentionally disabled. If you don’t use log deletion for this tenant, these errors are harmless but noisy. You can verify by sending a GET request to the compactor’s cache generation number endpoint:
Bash
```
curl - s - H "X- Scope- OrgID: <tenant>" http://compactor:3100/loki/api/v1/cache/generation_numbers
```
If the response is "deletion is not available for this tenant", the deletion API is not enabled for the tenant.
Enable deletion for the tenant if log deletion is required:
YAML
```
overrides:
  <tenant>:
    deletion_mode: filter- and- delete
```
Check compactor connectivity if the error includes a non- 403 status code or a connection error:
Bash
```
curl - s http://compactor:3100/ready
```
Verify the compactor address is correctly configured. Queriers and other components that use the cache generation loader need to reach the compactor:
YAML
```
compactor:
  compactor_address: http://compactor:3100
```

Properties:

Enforced by: Cache generation number loader (GenNumberLoader)
Retryable: Yes (the loader retries automatically every 5 minutes)
HTTP status: N/A (background process, not a request- time error)
Configurable per tenant: Yes (via deletion_mode in tenant overrides)

Ring and cluster communication errors

Ring errors occur when Loki components cannot properly communicate through the hash ring, which is used to distribute work across instances. The ring is fundamental to Loki’s distributed operation.

Error: Too many unhealthy instances in the ring

Error message:

too many unhealthy instances in the ring

Cause:

The ring contains too many unhealthy instances to satisfy the replication factor. For example, with a replication factor of 3, at least 3 healthy instances must be available.

Resolution:

Check the health of ring members:

curl -s http://loki:3100/ring | jq '.shards[] | select(.state != "ACTIVE")'

Restart unhealthy instances that are stuck in a bad state.
Scale up instances if there aren’t enough healthy members.
Check resource constraints (CPU, memory, disk) on unhealthy instances.

Properties:

Enforced by: Ring replication
Retryable: Yes (after instances recover)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Empty ring

Error message:

empty ring

Cause:

No instances are registered in the ring. This typically occurs during initial cluster startup, for example if your ingesters are OOM crashing, or due to misconfiguration.

Resolution:

Wait for instances to register during initial startup.
Check ingesters to make sure they are running.
Check that all instances can communicate over the configured ports.

Verify ring configuration across all components, especially memberlist configuration:

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3

Check KV store health (Consul, etcd, or memberlist):
Bash
```
# For memberlist
curl -s http://loki:3100/memberlist
```

Properties:

Enforced by: Ring operations
Retryable: Yes (after instances register)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Instance not found in the ring

Error message:

instance <id> not found in the ring

Cause:

A specific instance is expected to be in the ring but isn’t registered. This can happen after a restart if the instance hasn’t re-joined the ring yet.

Resolution:

Wait for the instance to re-register in the ring.
Check the instance’s logs for ring join failures.
Verify KV store connectivity from the instance.

Properties:

Enforced by: Ring operations
Retryable: Yes (after instance registers)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Instance owns no tokens

Error message:

this instance owns no tokens

Cause:

The instance has joined the ring but hasn’t claimed any tokens. Without tokens, the instance cannot receive any work. This can happen if:

The instance is still starting up
Token claim failed
The KV store update didn’t propagate

Resolution:

Wait for token assignment during startup.
Check the ring status for the instance: Open a browser and navigate to http://localhost:3100/ring. You should see the Loki Ring Status page.

OR
Bash
```
curl -s http://loki:3100/ring
```
Restart the instance if tokens are not assigned after startup completes.
Check KV store connectivity and health.

Properties:

Enforced by: Lifecycler readiness check
Retryable: Yes (after tokens are assigned)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Error talking to the KV store

Error message:

error talking to the KV store

Cause:

The instance cannot communicate with the key-value store used for ring state. The KV store (Consul, etcd, or memberlist) is required for ring coordination.

Resolution:

Check KV store health and connectivity:

# For Consul
curl http://consul:8500/v1/status/leader

# For etcd
etcdctl endpoint health

Verify network connectivity between Loki instances and the KV store.
Check firewall rules allow traffic on KV store ports.
For memberlist, verify that gossip ports are accessible between all instances:
YAML
```
memberlist:
  bind_port: 7946
  join_members:
    - loki-memberlist:7946
```

Properties:

Enforced by: Ring lifecycler
Retryable: Yes (after KV store recovery)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: No ring returned from the KV store

Error message:

no ring returned from the KV store

Cause:

The KV store responded but returned an empty or invalid ring descriptor. This can happen if the KV store was recently initialized or its data was cleared.

Resolution:

Wait for ring initialization during first startup.
Check if the KV store data was accidentally cleared.
Restart all ring members to re-register if the KV store was reset.

Properties:

Enforced by: Ring lifecycler
Retryable: Yes (after ring initialization)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Failed to join memberlist cluster

Error message:

failed to join memberlist cluster on startup

Or:

joining memberlist cluster failed

Cause:

The instance could not join the memberlist gossip cluster. Common causes:

Seed nodes are unreachable
DNS resolution failure for join addresses
Firewall blocking gossip ports
All existing members are down

Resolution:

Check that join members are reachable:

# Test connectivity to seed nodes
nc -zv loki-memberlist 7946

Verify DNS resolution for join addresses:
Bash
```
nslookup loki-memberlist
```

Check memberlist configuration:

memberlist:
  bind_port: 7946
  join_members:
    - loki-gossip-ring.loki.svc.cluster.local:7946

Ensure firewall rules allow UDP and TCP traffic on the gossip port (default 7946).
For Kubernetes, verify that the headless service for memberlist is configured correctly.

Properties:

Enforced by: Memberlist KV client
Retryable: Yes (automatic retries with backoff)
HTTP status: N/A (startup failure or degraded operation)
Configurable per tenant: No

Error: Re-joining memberlist cluster failed

Error message:

re-joining memberlist cluster failed

Cause:

After being disconnected from the memberlist cluster, the instance failed to rejoin. This can happen during network partitions or after prolonged network issues.

Resolution:

Check network connectivity between cluster members.
Verify other cluster members are healthy.
Restart the affected instance if automatic rejoin continues to fail.
Review network stability frequent re-joins indicate underlying network issues.

Properties:

Enforced by: Memberlist KV client
Retryable: Yes (automatic retries)
HTTP status: N/A (degraded operation)
Configurable per tenant: No

Component readiness errors

Readiness errors occur when Loki components are not ready to serve requests. These errors are returned by the /ready health check endpoint and prevent load balancers from routing traffic to unready instances.

Error: Application is stopping

Error message:

Application is stopping

Cause:

Loki is shutting down and no longer accepting new requests. This is normal during graceful shutdown.

Resolution:

Wait for the instance to restart if this is a rolling update.
Check if the shutdown is expected (maintenance, scaling down).
Review orchestrator logs (Kubernetes, systemd) if the shutdown is unexpected.

Properties:

Enforced by: Loki readiness handler
Retryable: Yes (after restart)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Some services are not running

Error message:

Some services are not Running:
<state>: <count>
<state>: <count>

For example:

Some services are not Running:
Starting: 1
Failed: 2

Cause:

One or more internal Loki services have failed to start or have stopped unexpectedly. The error message lists each service state with a count of services in that state.

Resolution:

Check Loki logs for errors from the listed services.
Verify configuration for the affected services.
Check resource availability (memory, disk, CPU).
Restart the instance if services are stuck.

Properties:

Enforced by: Loki service manager
Retryable: Yes (after services recover)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Ingester not ready

Error message:

Ingester not ready: <details>

When the ingester’s own state check fails, <details> contains the ingester state, giving the full message:

Ingester not ready: ingester not ready: <state>

Where <state> is the service state, for example Starting, Stopping, or Failed.

Cause:

The ingester is not in a ready state to accept writes or serve reads. The detail message indicates the specific reason, such as:

The ingester is still starting up and joining the ring (Starting)
The lifecycler is not ready (lifecycler error text)
The ingester is waiting for minimum ready duration after ring join

Resolution:

Wait for startup to complete - ingesters take time to join the ring and become ready.
Check ring membership: Open a browser and navigate to http://localhost:3100/ring. You should see the Loki Ring Status page.

OR
Bash
```
curl -s http://ingester:3100/ring
```
Review logs for startup errors.
Adjust the minimum ready duration if startup is too slow:
YAML
```
ingester:
  lifecycler:
    min_ready_duration: 15s
```

Properties:

Enforced by: Ingester readiness check
Retryable: Yes (after ingester becomes ready)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: No queriers connected to query frontend

Error message:

Query Frontend not ready: not ready: number of queriers connected to query-frontend is 0

Cause:

The query frontend has no querier workers connected. Without queriers, the frontend cannot process any queries. This typically occurs when:

Queriers are not yet started
Queriers cannot reach the frontend
gRPC connectivity issues between queriers and frontend

Resolution:

Check that queriers are running and healthy.
Verify querier configuration points to the correct frontend address:
YAML
```
frontend_worker:
  frontend_address: query-frontend:9095
```
Check gRPC connectivity between queriers and the frontend:
Bash
```
# Test gRPC port connectivity
nc -zv query-frontend 9095
```
Review querier logs for connection errors.

Properties:

Enforced by: Query frontend (v1) readiness check
Retryable: Yes (after queriers connect)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: No schedulers connected to frontend worker

Error message:

Query Frontend not ready: not ready: number of schedulers this worker is connected to is 0

Cause:

The query frontend worker has no active connections to any query scheduler. This prevents the frontend from dispatching queries.

Resolution:

Check that query schedulers are running and healthy.

Verify scheduler address configuration:

frontend_worker:
  scheduler_address: query-scheduler:9095

Check gRPC connectivity between the frontend and schedulers.
Review query scheduler logs for errors.

Properties:

Enforced by: Query frontend (v2) readiness check
Retryable: Yes (after schedulers connect)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

gRPC and message size errors

gRPC errors occur during inter-component communication. Loki components communicate using gRPC for ring coordination, query execution, and data transfer.

Error: Message size too large

Error message:

message size too large than max (<size> vs <max>)

Or for the decompressed body:

decompressed message size too large than max (<size> vs <max>)

Cause:

The compressed or decompressed body of an HTTP push request to the distributor exceeds the configured limit.

Default configuration:

distributor.max_recv_msg_size: 100MB (compressed request body limit)
distributor.max_decompressed_size: 5000MB (decompressed body limit, defaults to 50× max_recv_msg_size)

Resolution:

Increase the distributor receive message size limit:

distributor:
  max_recv_msg_size: 209715200      # 200MB compressed
  max_decompressed_size: 10737418240  # 10GB decompressed

Reduce push batch sizes in your log shipping client (Alloy, etc.) to send smaller individual requests.
Reduce the amount of data per request by lowering the batch size or flush interval in your client.

Properties:

Enforced by: Distributor push handler
Retryable: No (request must be smaller or limits increased)
HTTP status: 413 Request Entity Too Large (compressed), 400 Bad Request (decompressed)
Configurable per tenant: No

Error: Response larger than max message size

Error message:

response larger than the max message size (<size> vs <max>)

Cause:

A query result from the querier to the frontend exceeds the maximum allowed gRPC response size. This typically happens with queries that return very large result sets.

Default configuration:

server.grpc_server_max_send_msg_size: 4MB (gRPC server send limit on the querier)
querier.query_frontend_grpc_client.max_recv_msg_size: 100MB (gRPC client receive limit on the querier worker)

Resolution:

Reduce query scope to return fewer results:
- Add more specific label matchers
- Reduce the time range
- Lower the entries limit

Increase gRPC message size limits if needed. Apply these settings to querier nodes:

server:
  grpc_server_max_send_msg_size: 209715200   # 200MB

querier:
  query_frontend_grpc_client:
    max_recv_msg_size: 209715200             # 200MB

Properties:

Enforced by: Querier worker
Retryable: No (query scope or limits must change)
HTTP status: 413 Request Entity Too Large
Configurable per tenant: No

Error: Compressed message size exceeds limit

Error message:

compressed message size <size> exceeds limit <limit>

Cause:

The compressed body of an HTTP push request exceeds the distributor’s configured limit. This check runs after the request body has been fully read and validates the total compressed size against the configured maximum.

Default configuration:

distributor.max_recv_msg_size: 100MB

Resolution:

Reduce batch sizes in your log shipping client.
Split large batches into smaller, more frequent requests.

Increase the limit if needed:

distributor:
  max_recv_msg_size: 209715200   # 200MB

Properties:

Enforced by: Distributor push handler
Retryable: No (request must be smaller)
HTTP status: 400 Bad Request
Configurable per tenant: No

TLS and certificate errors

TLS errors occur when Loki or its clients cannot establish secure connections due to certificate issues.

Error: TLS certificate loading failed

Error message:

error loading ca cert: <path>

Or:

error loading client cert: <path>

Or:

error loading client key: <path>

Or:

failed to load TLS certificate <cert_path>,<key_path>

Cause:

Loki cannot load TLS certificates from the specified paths. Common causes:

Certificate files don’t exist at the configured paths
Permission issues prevent reading the files
Certificate or key format is invalid
Certificate and key don’t match

Resolution:

Verify certificate files exist and are readable:

ls -la /path/to/cert.pem /path/to/key.pem /path/to/ca.pem

Check file permissions (the Loki process must be able to read them).

Validate the certificate format:

openssl x509 -in /path/to/cert.pem -noout -text
openssl rsa -in /path/to/key.pem -check

Verify cert and key match:

openssl x509 -noout -modulus -in cert.pem | md5sum
openssl rsa -noout -modulus -in key.pem | md5sum
# Both should produce the same hash

Check your TLS configuration:

server:
  http_tls_config:
    cert_file: /path/to/cert.pem
    key_file: /path/to/key.pem
    client_ca_file: /path/to/ca.pem
  grpc_tls_config:
    cert_file: /path/to/cert.pem
    key_file: /path/to/key.pem
    client_ca_file: /path/to/ca.pem

Properties:

Enforced by: TLS configuration
Retryable: No (certificates must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: TLS configuration error

Error message:

error generating http tls config: <details>

Or:

error generating grpc tls config: <details>

Where <details> may include messages such as TLS version %q not recognized, cipher suite %q not recognized, or unknown TLS version: <version>.

Cause:

The TLS configuration is invalid. This can happen when:

An unsupported TLS version string is supplied
Cipher suite configuration is invalid
Client auth type is unrecognized

Resolution:

Review TLS settings for compatibility issues.
Use supported TLS versions by setting tls_min_version at the top level of the server block:
YAML
```
server:
  tls_min_version: VersionTLS12
```
Valid values are VersionTLS10, VersionTLS11, VersionTLS12, and VersionTLS13. There is no max_version setting; tls_min_version is the only version constraint.
Check cipher suite configuration if customized.

Properties:

Enforced by: TLS initialization
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

DNS resolution errors

DNS errors occur when Loki cannot resolve hostnames for service discovery or backend connections.

Error: DNS lookup timeout

Error message:

msg="failed to resolve server addresses" err="... DNS lookup timeout: [<address>] ..."

Cause:

DNS resolution exceeded the 5-second timeout when trying to resolve addresses for Loki service discovery or backend connections. This error is emitted by the index gateway and bloom gateway DNS discovery loops. The DNS lookup timeout: [<address>] string is the context cause embedded within the err field; the full address list is formatted as a Go slice (for example, [dns+loki-index-gateway.loki.svc.cluster.local:9095]).

Resolution:

Check DNS server availability and configuration.
Verify hostname resolution:
Bash
```
nslookup <hostname>
dig <hostname>
```

Use IP addresses as a workaround if DNS is unreliable:

# Instead of dns+hostname:port
memberlist:
  join_members:
    - 10.0.0.1:7946
    - 10.0.0.2:7946

For Kubernetes, ensure CoreDNS is healthy and headless services are configured correctly.

Properties:

Enforced by: Index gateway client, bloom gateway client DNS discovery loop
Retryable: Yes (DNS may recover)
HTTP status: N/A (connectivity failure)
Configurable per tenant: No

Scheduler and frontend errors

These errors relate to query scheduling, frontend workers, and queue management.

Error: Scheduler is not running

Error message:

scheduler is not running

Cause:

The query scheduler service is not in a running state. This can occur when:

The scheduler is starting up
The scheduler encountered a fatal error
The scheduler is shutting down

Resolution:

Check scheduler logs for startup errors or crashes.
Verify scheduler health:
Bash
```
curl -s http://scheduler:3100/ready
```
Check scheduler ring membership if using ring-based scheduling:
Bash
```
curl -s http://scheduler:3100/ring | jq
```

Properties:

Enforced by: Scheduler service
Retryable: Yes (wait for scheduler to become ready)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Too many outstanding requests

Error message:

too many outstanding requests

Cause:

The query queue has reached its maximum capacity. This indicates the system is overloaded with queries.

Resolution:

Scale out queriers to process queries faster:
YAML
```
querier:
  max_concurrent: 10
```
Increase queue capacity (with caution). The default is 32000; increase beyond that only if you have confirmed the system can handle the additional load. Note that increasing the queue is often necessary because of how many subqueries can be generated by large values for tsdb_max_query_parallelism. Generally it’s preferable to add more queriers and leave this setting unchanged.
YAML
```
query_scheduler:
  max_outstanding_requests_per_tenant: 64000
```
Rate limit queries at the client or load balancer level.
Optimize slow queries to reduce queue time.

Properties:

Enforced by: Query scheduler/frontend
Retryable: Yes
HTTP status: 429 Too Many Requests
Configurable per tenant: No

Error: Querying is disabled

Error message:

querying is disabled, please contact your Loki operator

Cause:

Query parallelism has been set to zero, effectively disabling all queries. This is typically done intentionally during maintenance.

Resolution:

Check the relevant parallelism setting for your index type. For TSDB indexes (the current default), tsdb_max_query_parallelism supersedes max_query_parallelism. Either value being set to zero triggers this error. Verify that both are greater than zero:
YAML
```
limits_config:
  max_query_parallelism: 32          # default; applies to non-TSDB schemas
  tsdb_max_query_parallelism: 128    # default; applies to TSDB schemas
```

Size tsdb_max_query_parallelism to your ingest volume. Typical values in production are in the range of 128–2048, proportional to the volume of logs ingested per day:

Daily ingest volume	Typical value
Low–moderate	128–256
High	512
Tens of TB/day	1024–2048

Account for the querier capacity this requires. Each unit of parallelism consumes one querier worker slot. With the default querier.max_concurrent of 4, the number of queriers needed to fully parallelize a single query is:
Bash
```
queriers needed = tsdb_max_query_parallelism / max_concurrent
```
For example, tsdb_max_query_parallelism: 2048 with max_concurrent: 4 requires 512 queriers to run one query fully in parallel. Production deployments supporting many tenants running large queries simultaneously commonly run thousands of queriers.
Contact your administrator if you don’t have access to change these settings.

Properties:

Enforced by: Query frontend
Retryable: No (configuration must change)
HTTP status: 429 Too Many Requests
Configurable per tenant: Yes

Error: No frontend address

Error message:

no frontend address

Cause:

The scheduler received a request from a frontend but no frontend address was provided for sending responses back.

Resolution:

Check frontend configuration to ensure the address is set:
YAML
```
frontend:
  address: query-frontend:9095
```
Verify gRPC connectivity between frontend and scheduler.

Properties:

Enforced by: Scheduler
Retryable: No (configuration issue)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Scheduler shutting down

Error message:

scheduler is shutting down

Cause:

The frontend scheduler worker detected that the scheduler is in shutdown mode and cannot accept new requests.

Resolution:

Wait for shutdown to complete and the scheduler to restart.
Check if this is expected (rolling update, maintenance).
Retry the request after the scheduler is healthy.

Properties:

Enforced by: Scheduler
Retryable: Yes (after scheduler restart)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Index gateway errors

Index gateway errors occur when queriers cannot communicate with index gateways for index lookups.

Error: Index gateway unhealthy in ring

Error message:

index-gateway is unhealthy in the ring

Cause:

The index gateway instance detects itself as unhealthy in the ring and refuses to process queries. This is a self-check: before handling tenant requests, the gateway verifies it appears in the set of healthy ring members.

Resolution:

Check index gateway health:
Bash
```
curl -s http://index-gateway:3100/ready
```
View the ring status: Open a browser and navigate to http://localhost:3100/ring. You should see the Loki Ring Status page.

OR
Bash
```
curl -s http://index-gateway:3100/ring
```
Check logs for errors preventing the gateway from becoming healthy.
Restart the index gateway if it’s stuck in an unhealthy state.

Properties:

Enforced by: Index gateway ring
Retryable: Yes (wait for gateway to become healthy)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: No index gateway instances found

Error message:

no index gateway instances found for tenant <tenant>

Cause:

No index gateway instances are available in the ring to serve the tenant’s request. This could be due to:

All index gateways are unhealthy
Shuffle sharding excludes this tenant
Ring is empty

Resolution:

Check if any index gateways are running:

curl -s http://index-gateway:3100/ring | jq '.shards | length'

Verify ring mode is configured if using shuffle sharding. The index gateway must run in ring mode and the per-tenant shard size must be set:
YAML
```
index_gateway:
  mode: ring

limits_config:
  index_gateway_shard_size: 3  # default = 0 (use all instances)
```
Scale up index gateways if needed.

Properties:

Enforced by: Index gateway client
Retryable: Yes
HTTP status: 500 Internal Server Error
Configurable per tenant: Yes (via index_gateway_shard_size in limits_config)

Error: Index client not initialized

Error message:

index client is not initialized likely due to boltdb-shipper not being used

Cause:

The index gateway was queried for operations that require the index client, but the client wasn’t initialized because the boltdb-shipper store isn’t configured.

Resolution:

Verify your schema config uses the correct index store:

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Check if the operation requires boltdb-shipper - some legacy operations may not be supported with TSDB.

Properties:

Enforced by: Index gateway
Retryable: No (configuration/schema issue)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Compactor and retention errors

Compactor errors occur during index compaction or retention enforcement.

Error: No chunks found in table

Error message:

no chunks found in table, please check if there are really no chunks and manually drop the table or see if there is a bug causing us to drop whole index table

Cause:

The compactor found an empty index table during retention processing. This could indicate:

All chunks in the table have expired
The table was never populated
Data corruption

Resolution:

Verify the table should be empty:

# Check if data exists for the time period
logcli query '{job=~".+"}' --from="<table-start-time>" --to="<table-end-time>" --limit=1

If the table is legitimately empty, manually delete it from object storage.
If data should exist, investigate potential data loss.

Properties:

Enforced by: Compactor retention
Retryable: No (requires manual intervention)
HTTP status: N/A (background process)
Configurable per tenant: No

Error: Delete request store not configured

Error message:

compactor.delete-request-store should be configured when retention is enabled

Cause:

Retention is enabled but no store is configured for tracking delete requests.

Resolution:

Configure the delete request store:

compactor:
  retention_enabled: true
  delete_request_store: s3

Or disable retention if not needed:
YAML
```
compactor:
  retention_enabled: false
```

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Max compaction parallelism invalid

Error message:

max compaction parallelism must be >= 1

Cause:

The compactor’s parallelism setting is configured to zero or a negative number.

Resolution:

Set a valid parallelism value:

compactor:
  max_compaction_parallelism: 1  # Must be >= 1

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Delete request not found

Error message:

could not find delete request with given id

Cause:

An attempt to cancel a delete request failed because no matching request exists.

Resolution:

List existing delete requests:

curl -s http://compactor:3100/loki/api/v1/delete | jq

Verify the delete request ID is correct.
Check if the request has already been processed and removed.

Properties:

Enforced by: Compactor API
Retryable: No
HTTP status: 404 Not Found
Configurable per tenant: No

Error: Retention is not enabled

Error message:

Retention is not enabled

Cause:

A delete request was submitted but retention is not enabled in the compactor configuration. Delete requests require retention to be enabled.

Resolution:

Enable retention in the compactor:

compactor:
  retention_enabled: true
  delete_request_store: s3

Restart the compactor after changing the configuration.

Properties:

Enforced by: Compactor delete request handler
Retryable: No (configuration must change)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Invalid delete request time format

Error message:

invalid start time: require unix seconds or RFC3339 format

Or:

invalid end time: require unix seconds or RFC3339 format

Cause:

The start or end time in a delete request is not in a valid format.

Resolution:

Use Unix seconds or RFC3339 format:

# Unix seconds
curl -X POST http://compactor:3100/loki/api/v1/delete \
  -H "X-Scope-OrgID: my-tenant" \
  -d "query={app=\"foo\"}" \
  -d "start=1704067200" \
  -d "end=1704153600"

# RFC3339
curl -X POST http://compactor:3100/loki/api/v1/delete \
  -H "X-Scope-OrgID: my-tenant" \
  -d "query={app=\"foo\"}" \
  -d "start=2024-01-01T00:00:00Z" \
  -d "end=2024-01-02T00:00:00Z"

Properties:

Enforced by: Compactor delete request handler
Retryable: No (request must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Delete request already processed

Error message:

deletion of request which is in process or already processed is not allowed

Cause:

An attempt was made to cancel a delete request that is already being processed or has completed processing.

Resolution:

Check the status of the delete request:

curl -s http://compactor:3100/loki/api/v1/delete \
  -H "X-Scope-OrgID: my-tenant" | jq

Submit a new delete request if you need to delete additional data.

Properties:

Enforced by: Compactor delete request handler
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Invalid max_interval for delete request

Error message:

invalid max_interval: valid time units are 's', 'm', 'h'

Or:

max_interval can't be greater than <configured-limit>

Or:

max_interval can't be greater than the interval to be deleted (<duration>)

Cause:

The max_interval parameter on a delete request has an invalid value, exceeds the configured delete_max_interval limit, or exceeds the time range of the delete request itself.

Resolution:

Use a valid time format with supported units (s, m, h):

curl -X POST http://compactor:3100/loki/api/v1/delete \
  -H "X-Scope-OrgID: my-tenant" \
  -d "query={app=\"foo\"}" \
  -d "start=1704067200" \
  -d "end=1704153600" \
  -d "max_interval=1h"

Properties:

Enforced by: Compactor delete request handler
Retryable: No (request must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Ruler errors

Ruler errors occur when evaluating alerting rules or recording rules.

Error: Invalid ruler evaluation config

Error message:

invalid ruler evaluation config: <details>

Cause:

The ruler evaluation mode configuration is invalid.

Resolution:

Use a valid evaluation mode:

ruler:
  evaluation:
    mode: local  # Or "remote"

Properties:

Enforced by: Ruler module initialization (initRuleEvaluator)
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Ruler remote write config conflict

Error message:

ruler remote write config: both 'client' and 'clients' options are defined; 'client' is deprecated, please only use 'clients'

Cause:

Both the deprecated client and the new clients configuration options are set for ruler remote write.

Resolution:

Remove the deprecated config and use clients:

ruler:
  remote_write:
    # Remove this:
    # client: {}

    # Use this instead:
    clients:
      primary:
        url: http://prometheus:9090/api/v1/write

Properties:

Enforced by: Ruler initialization (NewRuler)
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Remote write enabled but no URL configured

Error message:

remote-write enabled but no clients URL are configured

Or when multiple clients are configured in the clients map and one entry is missing a URL:

remote-write enabled but client '<name>' URL for tenant <client-id> is not configured

Cause:

Remote write is enabled for the ruler but no destination URL is configured. The first variant occurs when the clients map is empty. The second occurs when a named entry in the clients map has no url set; <client-id> is the map key for that entry, not a tenant ID.

Resolution:

Configure the remote write URL:

ruler:
  remote_write:
    enabled: true
    clients:
      primary:
        url: http://prometheus:9090/api/v1/write

Or disable remote write:

ruler:
  remote_write:
    enabled: false

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Rule result is not a vector or scalar

Error message:

rule result is not a vector or scalar

Cause:

A rule evaluation returned an unexpected result type. Both recording rules and alerting rules must produce vector or scalar results. A plain log-stream expression (one that returns log lines rather than a numeric metric) triggers this error in either rule type.

Resolution:

Check the rule expression returns a vector or scalar:

# Valid - returns vector:
record: my_metric
expr: sum(rate({job="app"}[5m] | json | level="error"))

# Invalid - returns logs (triggers error for both recording and alerting rules):
# record: my_metric
# expr: '{job="app"}'

Use aggregation functions to produce numeric results from log queries.

Properties:

Enforced by: Ruler evaluation
Retryable: No (rule must be fixed)
HTTP status: N/A (background process)
Configurable per tenant: No

Error: Ruler WAL closed

Error message:

WAL storage closed

Cause:

An operation was attempted on the ruler’s write-ahead log (WAL) after it was closed. This typically occurs during shutdown.

Resolution:

Wait for the ruler to restart if it’s restarting.
Check ruler logs for errors that caused unexpected WAL closure.
Verify disk space is available for WAL operations.

Properties:

Enforced by: Ruler WAL
Retryable: Yes (after ruler restart)
HTTP status: N/A
Configurable per tenant: No

Kafka integration errors

These errors occur when Loki is configured to use Kafka for ingestion.

Error: Missing Kafka address

Error message:

the Kafka address has not been configured

Cause:

Kafka ingestion is enabled but no Kafka broker address is configured.

Resolution:

Configure the Kafka address:

kafka_config:
  topic: loki-logs
  reader_config:
    address: kafka:9092
  writer_config:
    address: kafka:9092

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Missing Kafka topic

Error message:

the Kafka topic has not been configured

Cause:

Kafka ingestion is enabled but no topic name is configured.

Resolution:

Configure the Kafka topic:

kafka_config:
  topic: loki-logs
  reader_config:
    address: kafka:9092
  writer_config:
    address: kafka:9092

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Inconsistent SASL username and password

Error message:

both sasl username and password must be set

Cause:

Only one of the Simple Authentication and Security Layer (SASL) username or password is configured. Both must be set together.

Resolution:

Configure both username and password:

kafka_config:
  sasl_username: my-user
  sasl_password: ${KAFKA_PASSWORD}

Or remove both if SASL authentication is not required.

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Kafka enabled in distributor but not in ingester

Error message:

kafka is enabled in distributor but not in ingester

Cause:

Kafka is configured for the distributor but the ingester isn’t configured to read from Kafka. Both must be configured together.

Resolution:

Enable Kafka in both distributor and ingester:

distributor:
  kafka_writes_enabled: true

ingester:
  kafka_ingestion:
    enabled: true

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Bloom gateway errors

Bloom gateway errors occur when using bloom filters for query acceleration.

Error: Invalid bloom gateway addresses

Error message:

addresses requires a list of comma separated strings in DNS service discovery format with at least one item

Cause:

The bloom_gateway.client.addresses configuration field is empty or unset.

Resolution:

Configure valid addresses:
YAML
```
bloom_gateway:
  client:
    addresses: dns+bloom-gateway:9095
```
Valid formats:
- dns+hostname:port - DNS-based discovery
- host1:port,host2:port - Static list

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Request time range must span exactly one day

Error message:

request time range must span exactly one day

Cause:

Bloom gateway requests must be for exactly one day of data due to how bloom blocks are organized.

Resolution:

This is typically handled automatically by the bloom querier, which splits multi-day queries into per-day requests before sending them to the gateway. If you see this error:
- Check that the querier is properly configured
- Ensure queries are routed through the querier

Properties:

Enforced by: Bloom gateway
Retryable: No
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: From time must not be after through time

Error message:

from time must not be after through time

Cause:

The bloom gateway received a request where the start time (from) is later than the end time (through).

Resolution:

This indicates a malformed request reaching the bloom gateway. Verify that the client sending the request constructs time ranges correctly with from ≤ through.

Properties:

Enforced by: Bloom gateway
Retryable: No
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Write-ahead log (WAL) errors

WAL errors occur when the ingester cannot properly manage its write-ahead log.

Error: WAL is stopped

Error message:

wal is stopped

Cause:

An operation was attempted on the WAL after it was stopped. This typically occurs during shutdown or after a fatal error.

Resolution:

Check ingester health and logs for errors.
Verify disk space is available.
Restart the ingester if it’s in a bad state.

Properties:

Enforced by: Ingester WAL
Retryable: Yes (after ingester restart)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: Invalid checkpoint duration

Error message:

invalid checkpoint duration: <duration>

Cause:

The WAL checkpoint duration is set to an invalid value (likely zero or negative).

Resolution:

Set a valid checkpoint duration:

ingester:
  wal:
    checkpoint_duration: 5m

Properties:

Enforced by: Configuration validation
Retryable: No
HTTP status: N/A (startup failure)
Configurable per tenant: No

Ingester lifecycle errors

Ingester lifecycle errors occur during ingester startup, shutdown, or state transitions.

Error: Ingester is shutting down

Error message:

Ingester is shutting down

Cause:

The ingester is in the process of shutting down and is no longer accepting writes. This error (also known as ErrReadOnly) is returned when a push request arrives during graceful shutdown. During this period the ingester may still serve reads for data it holds in memory.

Resolution:

Configure clients to retry with backoff. The distributor will route to other healthy ingesters.
Wait for shutdown to complete and the new instance to start.
Check if shutdown is expected (rolling update, scaling event).
If unexpected, check orchestrator logs for OOM kills or health check failures.

Properties:

Enforced by: Ingester
Retryable: Partial. The distributor sends writes to all ingesters in the replication set in parallel and uses a quorum model. If the remaining ingesters meet the minimum success threshold, the overall write succeeds despite this error from a shutting-down ingester.
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: Ingester is stopping or already stopped

Error message:

Ingester is stopping or already stopped.

Cause:

The ingester’s shutdown management endpoint (POST /loki/api/v1/ingester/shutdown) was called when the ingester was not in a Running state. This happens when the endpoint is called a second time during an in-progress shutdown or after the ingester has already stopped. This error is returned by the shutdown endpoint, not by the log-write or query paths.

Resolution:

Do not call the shutdown endpoint again while a shutdown is already in progress.
Check orchestrator for duplicate shutdown signals or restart policies.
Investigate if the stop was unexpected (pod eviction, OOM, crash).

Properties:

Enforced by: Ingester shutdown endpoint
Retryable: No (the shutdown endpoint call itself is not retryable; wait for the ingester to restart before sending new writes)
HTTP status: 503 Service Unavailable (response from the shutdown endpoint)
Configurable per tenant: No

Error: Failed to start partition reader

Error message:

failed to start partition reader: <details>

Cause:

The ingester could not start its Kafka partition reader. This occurs when Kafka ingestion is enabled but the partition reader fails to initialize.

Resolution:

Check Kafka connectivity from the ingester.
Verify Kafka topic exists and the ingester has appropriate permissions.

Review Kafka configuration:

kafka:
  address: kafka:9092
  topic: loki-logs

Check Kafka broker health.

Properties:

Enforced by: Ingester startup
Retryable: No (configuration or infrastructure must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Failed to start partition ring lifecycler

Error message:

failed to start partition ring lifecycler: <details>

Cause:

The ingester could not start its Kafka partition ring lifecycler during startup. This is a separate component from the partition reader; it manages the ingester’s membership in the partition ring. This only occurs when Kafka ingestion is enabled.

Resolution:

Check Kafka connectivity from the ingester.
Verify the partition ring KV store (the store used for the partition ring) is reachable.
Review ingester logs for the wrapped error in <details>.
Check Kafka broker health and partition availability.

Properties:

Enforced by: Ingester startup
Retryable: No (configuration or infrastructure must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Lifecycler failed

Error message:

lifecycler failed: <details>

Cause:

The ingester’s lifecycler (which manages ring membership) encountered a fatal error. This prevents the ingester from participating in the ring.

Resolution:

Check KV store connectivity (Consul, etcd, or memberlist).
Review ingester logs for the specific lifecycler error.
Verify ring configuration is consistent across all ingesters.
Restart the ingester after fixing the underlying issue.

Properties:

Enforced by: Ingester lifecycler
Retryable: Yes (after fix and restart)
HTTP status: N/A (internal failure)
Configurable per tenant: No

Pattern ingester errors

Pattern ingester errors occur when using the pattern ingester for automatic log pattern detection.

Error: Pattern ingester replication factor must be 1

Error message:

pattern ingester replication factor must be 1

Cause:

The pattern ingester is configured with a replication factor other than 1. Currently, the pattern ingester only supports a replication factor of 1.

Resolution:

Set the replication factor to 1:

pattern_ingester:
  lifecycler:
    ring:
      replication_factor: 1

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Pattern ingester retain-for too short

Error message:

retain-for (<duration>) must be greater than or equal to chunk-duration (<duration>)

Cause:

The pattern ingester’s retain_for duration is shorter than max_chunk_age, which would cause data loss.

Resolution:

Increase the retain-for duration to be at least as long as max_chunk_age:

pattern_ingester:
  retain_for: 15m   # Must be >= max_chunk_age
  max_chunk_age: 5m

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Pattern ingester chunk-duration too short

Error message:

chunk-duration (<duration>) must be greater than or equal to sample-interval (<duration>)

Cause:

The pattern ingester’s max_chunk_age is shorter than pattern_sample_interval. Chunks must span at least one sample interval to hold any data.

Resolution:

Increase max_chunk_age to be at least as long as pattern_sample_interval:

pattern_ingester:
  max_chunk_age: 1h     # Must be >= pattern_sample_interval (default: 1h)
  pattern_sample_interval: 10s  # default: 10s

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Error: Pattern ingester volume threshold out of range

Error message:

volume_threshold (<value>) must be between 0 and 1

Cause:

The volume_threshold value is outside the valid range of 0 to 1. This setting controls what fraction of log volume the pattern ingester tracks — only patterns representing the top X% of log volume are persisted.

Resolution:

Set volume_threshold to a value between 0 and 1 (default is 0.99):
YAML
```
pattern_ingester:
  volume_threshold: 0.99
```

Properties:

Enforced by: Configuration validation
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

API parameter errors

These errors occur when API requests contain invalid parameters.

Error: Invalid direction

Error message:

invalid direction '<value>'

Cause:

The direction query parameter contains an invalid value.

Resolution:

Use a valid direction value:
- forward - Oldest to newest
- backward - Newest to oldest (default)
Bash
```
curl "http://loki:3100/loki/api/v1/query_range?query={job=\"app\"}&direction=forward"
```

Properties:

Enforced by: API handler
Retryable: No (request must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Limit must be a positive value

Error message:

limit must be a positive value

Cause:

The limit parameter is zero or negative.

Resolution:

Provide a positive limit:

curl "http://loki:3100/loki/api/v1/query_range?query={job=\"app\"}&limit=100"

Properties:

Enforced by: API handler
Retryable: No (request must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: End timestamp must not be before start time

Error message:

end timestamp must not be before or equal to start time

Cause:

The query’s end time is before or equal to its start time.

Resolution:

Ensure end time is after start time:

curl "http://loki:3100/loki/api/v1/query_range?\
query={job=\"app\"}&\
start=2024-01-01T00:00:00Z&\
end=2024-01-02T00:00:00Z"

Properties:

Enforced by: API handler
Retryable: No (request must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Delay for tailing too large

Error message:

delay_for can't be greater than <max>

Cause:

The delay_for parameter for tailing queries exceeds the maximum allowed value.

Resolution:

Reduce the delay_for value:
Bash
```
curl "http://loki:3100/loki/api/v1/tail?query={job=\"app\"}&delay_for=5"
```
The maximum value is typically 5 seconds.

Properties:

Enforced by: API handler
Retryable: No (request must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Query filtering requires compactor address

Error message:

query filtering for deletes requires 'compactor_grpc_address' or 'compactor_address' to be configured

Cause:

Query-time filtering for delete requests is enabled but Loki doesn’t know how to reach the compactor to retrieve active delete requests.

Resolution:

Configure the compactor address:

compactor:
  compactor_grpc_address: compactor:9095

Or use the HTTP address:

compactor:
  compactor_address: http://compactor:3100

Properties:

Enforced by: Module initialization
Retryable: No (configuration must be fixed)
HTTP status: N/A (startup failure)
Configurable per tenant: No

Troubleshoot log ingestion (WRITE)

Thu, 09 Apr 2026 00:47:14 +0000

Troubleshoot log ingestion (WRITE)

This guide helps you troubleshoot errors that occur when ingesting logs into Loki and writing logs to storage. When Loki rejects log ingestion requests, it’s typically due to exceeding rate limits, violating validation rules, or encountering storage issues.

Before you begin, ensure you have the following:

Access to Grafana Loki logs and metrics
Understanding of your log ingestion pipeline (Alloy or custom clients)
Permissions to configure limits and settings if needed

Monitoring ingestion errors

All ingestion errors are tracked using Prometheus metrics:

loki_discarded_samples_total - Count of discarded log samples by reason
loki_discarded_bytes_total - Volume of discarded log data by reason

The reason label on these metrics indicates the specific error type. Set up alerts on these metrics to detect ingestion problems.

Rate limit errors

Rate limits protect Loki from being overwhelmed by excessive log volume. When rate limits are exceeded, Loki returns HTTP status code 429 Too Many Requests.

Error: `rate_limited`

Error message:

ingestion rate limit exceeded for user <tenant> (limit: <limit> bytes/sec) while attempting to ingest <lines> lines totaling <bytes> bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased

Cause:

The tenant has exceeded their configured ingestion rate limit. This is a global per-tenant limit enforced by the distributor.

Default configuration:

ingestion_rate_mb: 4 MB/sec
ingestion_burst_size_mb: 6 MB
ingestion_rate_strategy: “global” (shared across all distributors)

Resolution:

Increase rate limits (if you have sufficient cluster resources):
YAML
```
limits_config:
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 12
```
Reduce log volume by:
- Filtering unnecessary log lines using Alloy processing stages
- Collecting logs from fewer targets
- Sampling high-volume streams
- Using Alloy’s rate limiting and filtering capabilities
Switch to local strategy (if using single-region deployment):
YAML
```
limits_config:
  ingestion_rate_strategy: local
```
Note
This multiplies the effective limit by the number of distributor replicas.

Properties:

Enforced by: Distributor
Retryable: Yes
HTTP status: 429 Too Many Requests
Configurable per tenant: Yes

Error: `stream_limit`

Error message:

maximum active stream limit exceeded when trying to create stream <stream_labels>, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: <tenant>

Cause:

The tenant has reached the maximum number of active streams. Active streams are held in memory on ingesters, and excessive streams can cause out-of-memory errors.

Default configuration:

max_global_streams_per_user: 5000 (globally)
max_streams_per_user: 0 (no local limit per ingester)
Active stream window: chunk_idle_period (default: 30 minutes)

Terminology:

Active stream: A stream that has received logs within the chunk_idle_period (default: 30 minutes)

Resolution:

Reduce stream cardinality by:
- Using fewer unique label combinations by removing unnecessary labels
- Avoiding high-cardinality labels (IDs, timestamps, UUIDs)
- Using structured metadata instead of labels for high-cardinality data
Increase stream limits (if you have sufficient memory):
YAML
```
limits_config:
  max_global_streams_per_user: 10000
```

Note
Do not increase stream limits to accommodate high cardinality labels, this can result in Loki flushing extremely high numbers of small files which will make for extremely poor query performance. As volume and the size of the infrastructure being monitored increase, it would be expected to increase the stream limit. However even for hundreds of TBs per day of logs you should avoid exceeding 300,000 max global streams per user.

Properties:

Enforced by: Ingester
Retryable: Yes
HTTP status: 429 Too Many Requests
Configurable per tenant: Yes

Validation errors

Validation errors occur when log data doesn’t meet Loki’s requirements. These return HTTP status code 400 Bad Request and are not retryable.

Error: `line_too_long`

Error message:

max entry size <max_size> bytes exceeded for stream <stream_labels> while adding an entry with length <entry_size> bytes

Cause:

A log line exceeds the maximum allowed size.

Default configuration:

max_line_size: 256 KB
max_line_size_truncate: false

Resolution:

Truncate long lines instead of discarding them:

limits_config:
  max_line_size: 256KB
  max_line_size_truncate: true

Increase the line size limit (not recommended above 256KB):
YAML
```
limits_config:
  max_line_size: 256KB
```
Warning
Loki was built as a large multi-user, multi-tenant database and as such this limit becomes very important to maintain stability and performance with many users query the database simultaneously. We strongly recommend against increasing the max_line_size, doing so will make it very difficult to provide consistent query performance and stability of the system without having to throw extremely large amounts of memory and/or increasing the GRPC message size limits to really high levels, both of which will likely lead to poorer performance and worse experiences.
Filter or truncate logs before sending using Alloy processing stages:
Alloy
```
stage.replace {
  expression = "^(.{10000}).*$"
  replace    = "$1"
}
```

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: `invalid_labels`

Error message:

error parsing labels <labels> with error: <parse_error>

Cause:

Label names or values contain invalid characters or don’t follow Prometheus naming conventions:

Label names must match regex: [a-zA-Z_][a-zA-Z0-9_]*
Label names cannot start with __ (reserved for Grafana internal use)
Labels must be valid UTF-8
Properly escape special characters in label values
Ensure label syntax follows PromQL format: {key="value"}

Default configuration:

This validation is always enabled and cannot be disabled.

Resolution:

Fix label names in your log shipping configuration:

# Incorrect
labels:
  "123-app": "value"      # Starts with number
  "app-name": "value"     # Contains hyphen
  "__internal": "value"   # Reserved prefix

# Correct
labels:
  app_123: "value"
  app_name: "value"
  internal_label: "value"

Use Alloy’s relabeling stages to fix labels:

stage.label_drop {
  values = ["invalid_label"]
}

stage.labels {
  values = {
    app_name = "",
  }
}

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: `missing_labels`

Error message:

error at least one label pair is required per stream

Cause:

A log stream was submitted without any labels. Loki requires at least one label pair for each log stream.

Default configuration:

This validation is always enabled and cannot be disabled. Every stream must have at least one label.

Resolution:

Ensure all log streams have at least one label. Update your log shipping configuration to include labels:

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.source.file "logs" {
  targets = [
    {
      __path__ = "/var/log/*.log",
      job      = "myapp",
      environment = "production",
    },
  ]
  forward_to = [loki.write.default.receiver]
}

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: `out_of_order` / `too_far_behind`

These errors occur when log entries arrive in the wrong chronological order.

Error messages:

When unordered_writes: false: entry out of order
When unordered_writes: true: entry too far behind, entry timestamp is: <timestamp>, oldest acceptable timestamp is: <cutoff>

Cause:

Logs are being ingested with timestamps that violate Loki’s ordering constraints:

With unordered_writes: false: Any log older than the most recent log in the stream is rejected
With unordered_writes: true (default): Logs older than half of max_chunk_age from the newest entry are rejected

Default configuration:

unordered_writes: true (since Loki 2.4)
max_chunk_age: 2 hours
Acceptable timestamp window: 1 hour behind the newest entry (half of max_chunk_age)

Resolution:

Ensure timestamps are assigned correctly:
- Use a timestamp parsing stage in Alloy if logs have embedded timestamps
- Avoid letting Alloy assign timestamps at read time for delayed log delivery
Alloy
```
stage.timestamp {
  source = "timestamp_field"
  format = "RFC3339"
}
```
Check for clock skew between log sources and Loki. Ensure clocks are synchronized across your infrastructure.

Properties:

Enforced by: Ingester
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: No (for max_chunk_age)

Error: `greater_than_max_sample_age`

Error message:

entry for stream <stream_labels> has timestamp too old: <timestamp>, oldest acceptable timestamp is: <cutoff>

Cause:

The log entry’s timestamp is older than the configured maximum sample age. This prevents ingestion of very old logs.

Default configuration:

reject_old_samples: true
reject_old_samples_max_age: 168 hours (7 days)

Resolution:

Increase the maximum sample age:

limits_config:
  reject_old_samples_max_age: 336h  # 14 days

Fix log delivery delays causing old timestamps.
For historical log imports, temporarily disable the check per tenant.
Check for clock skew between log sources and Loki. Ensure clocks are synchronized across your infrastructure.
Disable old sample rejection (not recommended):
YAML
```
limits_config:
  reject_old_samples: false
```

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: `too_far_in_future`

Error message:

entry for stream <stream_labels> has timestamp too new: <timestamp>

Cause:

The log entry’s timestamp is further in the future than the configured grace period allows.

Default configuration:

creation_grace_period: 10 minutes

Resolution:

Check for clock skew between log sources and Loki. Ensure clocks are synchronized across your infrastructure.
Verify application timestamps Validate timestamp generation in your applications.
Verify timestamp parsing in your log shipping configuration.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: `max_label_names_per_series`

Error message:

entry for stream <stream_labels> has <count> label names; limit <limit>

Cause:

The stream has more labels than allowed.

Default configuration:

max_label_names_per_series: 15

Resolution:

Reduce the number of labels by:
- Using structured metadata for high-cardinality data
- Removing unnecessary labels
- Combining related labels

Warning
We strongly recommend against increasing max_label_names_per_series, doing so creates a larger index which hurts query performance as well as opens the door for cardinality explosions. You should be able to categorize your logs with 15 labels or typically much less. In all our years of running Loki, out of thousands of requests to increase this value, the number of valid exceptions we have seen can be counted on one hand.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: `label_name_too_long`

Error message:

stream <stream_labels> has label name too long: <label_name>

Cause:

A label name exceeds the maximum allowed length.

Default configuration:

max_label_name_length: 1024 bytes

Resolution:

Shorten label names in your log shipping configuration to under the configured limit. You can use abbreviations or shorter descriptive names.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: `label_value_too_long`

Error message:

stream <stream_labels> has label value too long: <label_value>

Cause:

A label value exceeds the maximum allowed length.

Default configuration:

max_label_value_length: 2048 bytes

Resolution:

Shorten label values in your log shipping configuration to under the configured limit. You can use hash values for very long identifiers.
Use structured metadata for long values instead of labels.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: `duplicate_label_names`

Error message:

stream <stream_labels> has duplicate label name: <label_name>

Cause:

The stream has two or more labels with identical names.

Default configuration:

This validation is always enabled and cannot be disabled. Duplicate label names are never allowed.

Resolution:

Remove duplicates Remove duplicate label definitions in your log shipping configuration to ensure unique label names per stream.
Check clients Check your ingestion pipeline for label conflicts.
Verify processing Verify your label processing and transformation rules.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: `request_body_too_large`

Error message:

Request body too large: <size> bytes, limit: <limit> bytes

Cause:

The HTTP compressed push request body exceeds the configured limit in your gateway/reverse proxy or service that fronts Loki after decompression.

Default configuration:

Loki server max request body size: Configured at the HTTP server level
Nginx gateway (Helm): gateway.nginxConfig.clientMaxBodySize (default: 4 MB)
Alloy default batch size: 1 MB

Resolution:

Split large batches into smaller, more frequent requests.
Increase batch limit Increase the allowed body size on your gateway/reverse proxy. For example, in the Helm chart set gateway.nginxConfig.clientMaxBodySize; default is 4M.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 413 Request Entity Too Large
Configurable per tenant: No

Storage errors

Storage errors occur when Loki cannot write data to its storage backend or the Write-Ahead Log (WAL).

Error: Disk space full

Error message:

no space left on device

Cause:

Local storage has run out of space. This can affect ingester WAL, chunk cache, or temporary files.

Default configuration:

Disk space is not limited by Loki configuration; it depends on your infrastructure provisioning.

Resolution:

Free up disk space on Loki instances.
Configure retention policies to remove old data.
Scale storage capacity by adding more disk space.
Monitor disk usage and set up alerts before reaching critical levels.
Review compactor settings to ensure old data is being cleaned up.

Properties:

Enforced by: Operating system
Retryable: Yes (after freeing space)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: Chunk write failure

Error message:

failed to store chunks: storage unavailable

Common causes and errors:

S3/Object Storage errors:
- NoSuchBucket: Storage bucket doesn’t exist
- AccessDenied: Invalid credentials or permissions
- RequestTimeout: Network or storage latency issues

Resolution:

Verify storage configuration:

storage_config:
  aws:
    s3: s3://region/bucket-name
    s3forcepathstyle: true

Check credentials and permissions:
- Ensure service account has write permissions
- Verify IAM roles for S3
- Check API keys for cloud storage
Monitor storage health:
- Check object storage availability
Review storage metrics:
- loki_ingester_chunks_flushed_total
- loki_ingester_chunks_flush_errors_total

Increase retries and timeouts:

storage_config:
  aws:
    s3:
      http_config:
        idle_conn_timeout: 90s
        response_header_timeout: 0s

Properties:

Enforced by: Ingester (during chunk flush)
Retryable: Yes (with backoff)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: WAL disk full

Error logged:

Error writing to WAL, disk full, no further messages will be logged for this error

Metric: loki_ingester_wal_disk_full_failures_total

Cause:

The disk where the WAL is stored has run out of space. When this occurs, Loki continues accepting writes but doesn’t log them to the WAL, losing durability guarantees.

Default configuration:

WAL enabled by default
WAL location: configured by -ingester.wal-dir
Checkpoint interval: ingester.checkpoint-duration (default: 5 minutes)

Resolution:

Increase disk space for the WAL directory.

Monitor disk usage and set up alerts:

loki_ingester_wal_disk_full_failures_total > 0

Reduce log volume to decrease WAL growth.
Check WAL checkpoint frequency:
YAML
```
ingester:
  checkpoint_duration: 5m
```
Verify WAL cleanup is working - old segments should be deleted after checkpointing.

Properties:

Enforced by: Ingester
Retryable: N/A (writes are accepted but not persisted to WAL)
HTTP status: N/A (writes succeed, but durability is compromised)
Configurable per tenant: No

Note
The WAL sacrifices durability for availability - it won’t reject writes when the disk is full. After disk space is restored, durability guarantees resume. Use metric loki_ingester_wal_disk_usage_percent to monitor disk usage.

Error: WAL corruption

Error message:

encountered WAL read error, attempting repair

Metric: loki_ingester_wal_corruptions_total

Cause:

The WAL has become corrupted, possibly due to:

Disk errors
Process crashes during write
Filesystem issues
Partial deletion of WAL files

Default configuration:

WAL enabled by default
Automatic repair attempted on startup
WAL location: configured by -ingester.wal-dir

Resolution:

Monitor for corruption:

increase(loki_ingester_wal_corruptions_total[5m]) > 0

Automatic recovery: Loki attempts to recover readable data and continues starting.
Investigate root cause:
- Check disk health
- Review system logs for I/O errors
- Verify filesystem integrity

Properties:

Enforced by: Ingester (on startup)
Retryable: N/A (Loki attempts automatic recovery)
HTTP status: N/A (occurs during startup, not request handling)
Configurable per tenant: No

Note
Loki prioritizes availability over complete data recovery. The replication factor provides redundancy if one ingester loses WAL data. Recovered data may be incomplete if corruption is severe.

Network and connectivity errors

These errors occur due to network issues or service unavailability.

Error: Connection refused

Error message:

connection refused when connecting to loki:3100

Cause:

The Loki service is unavailable or not listening on the expected port.

Default configuration:

Loki HTTP listen port: 3100 (-server.http-listen-port)
Loki gRPC listen port: 9095 (-server.grpc-listen-port)

Resolution:

Verify Loki is running and healthy.
Check network connectivity between client and Loki.
Confirm the correct hostname and port configuration.
Review firewall and security group settings.
Check for CPU starvation as a Loki pod that is overwhelmed could fail to respond to a request.

Properties:

Enforced by: Network/OS
Retryable: Yes (after service recovery)
HTTP status: N/A (connection-level error)
Configurable per tenant: No

Error: Context deadline exceeded

Error message:

context deadline exceeded

Cause:

Requests are timing out due to slow response times or network issues.

Default configuration:

Alloy default timeout: 10 seconds
Loki server timeout: -server.http-server-write-timeout (default: 30s)

Resolution:

Increase timeout values in your ingestion client.
Check Loki performance and resource utilization.
Review network latency between client and server.
Scale Loki resources if needed.

Properties:

Enforced by: Client/Server
Retryable: Yes
HTTP status: 504 Gateway Timeout (or client-side timeout)
Configurable per tenant: No

Error: Service unavailable

Error message:

service unavailable (503)

Cause:

Loki is temporarily unable to handle requests due to high load or maintenance. This can occur when ingesters are unhealthy or the ring is not ready.

Default configuration:

Ingester ring readiness: Requires minimum healthy ingesters based on replication factor
Distributor: Returns 503 when no healthy ingesters available

Resolution:

Implement retry logic with exponential backoff in your client.
Scale Loki ingester instances to handle load.
Check for resource constraints (CPU, memory, storage).
Review ingestion patterns for sudden spikes.

Properties:

Enforced by: Distributor/Gateway
Retryable: Yes
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Structured metadata errors

These errors occur when using structured metadata incorrectly.

Error: Structured metadata disabled

Error message:

stream <stream_labels> includes structured metadata, but this feature is disallowed. Please see limits_config.allow_structured_metadata or contact your Loki administrator to enable it

Cause:

Structured metadata is disabled in the Loki configuration. This feature must be explicitly enabled.

Default configuration:

allow_structured_metadata: true (enabled by default in recent versions)

Resolution:

Enable structured metadata:

limits_config:
  allow_structured_metadata: true

Move metadata to regular labels if structured metadata isn’t needed.
Contact your Loki administrator to enable the feature.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Structured metadata too large

Error message:

stream '{job="app"}' has structured metadata too large: '70000' bytes, limit: '65536' bytes. Please see limits_config.max_structured_metadata_size or contact your Loki administrator to increase it

Cause:

The structured metadata size exceeds the configured limit.

Default configuration:

max_structured_metadata_size: 64 KB

Resolution:

Reduce the size of structured metadata by removing unnecessary fields.

Increase the limit:

limits_config:
  max_structured_metadata_size: 128KB

Use compression or abbreviations for metadata values.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Too many structured metadata entries

Error message:

stream '{job="app"}' has too many structured metadata labels: '150', limit: '128'. Please see limits_config.max_structured_metadata_entries_count or contact your Loki administrator to increase it

Cause:

The number of structured metadata entries exceeds the limit.

Default configuration:

max_structured_metadata_entries_count: 128

Resolution:

Reduce the number of structured metadata entries by consolidating fields.

Increase the limit:

limits_config:
  max_structured_metadata_entries_count: 256

Combine related metadata into single entries using JSON or other formats.

Properties:

Enforced by: Distributor
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Blocked ingestion errors

These errors occur when ingestion is administratively blocked.

Ingestion blocked

Error message:

ingestion blocked for user <tenant> until <time> with status code 260

Or for policy-specific blocks:

ingestion blocked for user <tenant> (policy: <policy>) until <time> with status code 260

Cause:

Ingestion has been administratively blocked for the tenant, typically due to policy violations, billing issues, or maintenance. Status code 260 is a custom Loki status code for blocked ingestion.

Default configuration:

blocked_ingestion_status_code: 260 (custom status code)
Blocks are configured per tenant or per policy in runtime overrides

Resolution:

Wait until the block expires if a time limit is set.
Contact your Loki administrator to understand the block reason.
Review tenant usage patterns and policies.
Check for policy-specific blocks if using ingestion policies.

Properties:

Enforced by: Distributor
Retryable: No (until block is lifted)
HTTP status: 260 (custom) or configured status code
Configurable per tenant: Yes

Troubleshooting workflow

Follow this workflow when investigating ingestion issues:

Check metrics:

# See which errors are occurring
sum by (reason) (rate(loki_discarded_samples_total[5m]))

# Identify affected tenants
sum by (tenant, reason) (rate(loki_discarded_bytes_total[5m]))

Review logs from distributors and ingesters:

# Enable write failure logging per tenant
limits_config:
  limited_log_push_errors: true

Test push endpoint:

curl -H "Content-Type: application/json" \
     -XPOST http://loki:3100/loki/api/v1/push \
     --data-raw '{"streams":[{"stream":{"job":"test"},"values":[["'"$(date +%s)000000000"'","test log line"]]}]}'

Check tenant limits:

curl http://loki:3100/loki/api/v1/user_stats

Review configuration for affected tenant or global limits.
Monitor system resources:
- Ingester memory usage
- Disk space for WAL
- Storage backend health
- Network connectivity

Additional resources

Troubleshoot logs Drilldown

Thu, 09 Apr 2026 00:47:14 +0000

Troubleshoot Logs Drilldown

This page addresses common issues when getting started and using Grafana Logs Drilldown.

Grafana Explore Logs is installed by default in Grafana versions v11.3.0 through v11.5.

Grafana Logs Drilldown is installed by default in Grafana versions v11.6.11 and later.

In Grafana v12 and later, the Drilldown menu includes all Drilldown apps by default.

For more information about the name change for this feature, see this blog post.

If you do not see Logs Drilldown under either name, then check to make sure you have the Grafana Logs Drilldown plugin installed and configured.

Note
Your instance needs internet connection in order to download the Logs Drilldown plugin. If you are working in an offline environment, you can download the Logs Drilldown plugin separately and add it to your Grafana /plugins repository.

Ensure Loki is properly configured

To use Grafana Logs Drilldown, you need to have Loki properly configured. You can find full instructions in Access or install Grafana Logs Drilldown.

Service selection errors

This section covers errors on the service selection page.

There are no services

If everything is presented as an unknown_service when you access Grafana Logs Drilldown, you can try the following fixes:

Ensure the Volume API is enabled by setting the volume_enabled configuration value in Loki. Enabled by default in Loki 3.1 and later.
Specify the label to use to identify services by setting the discover_service_name configuration value in Loki.

No logs found in [label]

This message appears when the selected label has no log volume data for the current time range.

Try the following fixes:

Expand your time range to find when logs were last recorded for this label.
Select a different label or service that has active log data.
Verify that logs are being ingested for this label by checking your log collection pipeline.

Log volume has not been configured

This error indicates that Loki’s volume API is not enabled, which Logs Drilldown requires to display service volumes.

To fix this, enable volume in your Loki configuration:

limits_config:
  volume_enabled: true

Refer to the Loki configuration documentation for more details.

No service matched your search

This message appears when searching for services on the service selection page and no results match your search term.

Try the following fixes:

Check your search term for typos.
Try a partial match or different search term.
Clear the search to see all available services.

There are no detected levels

If you do not see detected_level values in Grafana Logs Drilldown, you can try the following fixes:

Ensure level detection is enabled by setting the discover_log_levels configuration value. Enabled by default in Loki 3.1 and later.

Label and field errors

This section covers errors related to labels and fields in Logs Drilldown.

There are no labels

If you do not see any labels in Grafana Logs Drilldown, you can try the following fixes:

Ensure your collector is properly configured to attach them.

To learn more about Labels, refer to the Loki labels documentation.

No labels selected

This message appears when you haven’t selected any labels to filter your logs. Logs Drilldown requires at least one label filter to display results.

To fix this:

Select a service or other primary label from the service selection page.
If you removed all labels, click the Reset filters button to restore the default selection.

Invalid labels selected

This error occurs when your label filters are in an invalid state, typically when all remaining filters use negative matching (not equal, not regex match).

Try the following fixes:

Ensure you have at least one label filter with positive (inclusive) matching.
Click the Reset filters button to restore a valid label selection.
Add a new label filter that includes (rather than excludes) logs.

The labels are not available at this moment

This warning appears when Logs Drilldown temporarily cannot fetch label information from Loki.

Try the following fixes:

Wait a moment and refresh the page.
Try selecting a different time range.
Check your Loki data source connection in Grafana’s data source settings.

No labels/fields match these filters

This message appears when your current filters don’t match any available labels or fields.

Try the following fixes:

Clear your current filters using the Clear filters button.
Broaden your time range.
Check that your label filter values are correct and exist in your log data.

We did not find any fields for the given time range

This message indicates that no detected fields (structured metadata or parsed fields) were found for your current selection and time range.

Try the following fixes:

Expand your time range to include more log data.
Verify that your logs contain structured data (JSON, logfmt, or other parsable formats).
Check that structured metadata is enabled in your Loki configuration by setting allow_structured_metadata: true.

Default fields tab not appearing

If the Default fields tab does not appear in the Logs Drilldown plugin settings, or if it displays an unsupported message, check the following:

Ensure you are running Grafana 12.4 or later. Default fields requires this minimum version.
Verify that the kubernetesLogsDrilldown feature flag is enabled in your Grafana configuration.
Confirm that you have the Org Admin role in your organization.

For more information about configuring Default fields, refer to Configure Logs Drilldown.

There are no color levels

Color coding for log severity levels is a setting in Loki. You must have discover_log_levels: true in your Loki configuration file.

Query and results errors

This section covers common error messages you may encounter when querying logs.

No logs match your search

If you see the message “No logs match your search. Please review your filters or try a different time range.” it means your current combination of filters and time range returns no results.

Try the following fixes:

Expand the time range to include a larger window of logs.
Review your label filters and remove any that may be too restrictive.
Check your line filter (search text) for typos or overly specific patterns.
If you have active pattern filters (include or exclude), try clearing them to see if logs appear.

Tip
You can click the Clear filters button that appears with the error message to reset all filters and start fresh.

Logs could not be retrieved due to invalid filter parameters

This error occurs when the query contains invalid syntax, typically from a malformed regular expression in a line filter.

Try the following fixes:

Check your line filter for invalid regex syntax. Loki uses RE2 syntax for regular expressions.
If you’re using special characters like (, ), [, ], {, }, ., *, +, ?, ^, $, |, or \, ensure they are properly escaped.
Toggle off the regex mode in the line filter if you’re searching for literal text.

The response is too large to process

This error appears when Loki returns more data than it can handle in a single response.

Try the following fixes:

Narrow your time range to reduce the amount of data returned.
Add more specific label filters to target a smaller subset of logs.
Use a line filter to search for specific text, which reduces the response size.
If you’re an administrator, consider adjusting the max_recv_msg_size in your Loki server configuration.

Max entries limit per query exceeded

This error means your query is trying to return more log lines than the configured limit allows. Loki has a default limit on the number of log entries that can be returned in a single query.

Try the following fixes:

Reduce the Line limit setting in the Logs Drilldown interface. You can find this option next to the line filter on the Logs tab.
Narrow your time range to return fewer results.
Add more specific filters to reduce the number of matching logs.
If you’re an administrator and need higher limits, adjust the max_entries_limit_per_query in your Loki configuration.

Max series limit exceeded

If you see “Max series limit exceeded” when viewing a field breakdown, it means the query returned more unique label value combinations than Loki allows.

Try the following fixes:

Reduce the time range to lower the number of unique series.
Add additional filters to narrow down the results before viewing the field breakdown.
Choose a different field with fewer unique values to break down.
If you’re an administrator, you can increase the max_query_series limit in your Loki configuration, though be aware this may impact query performance.

Tip
Fields with high cardinality (many unique values) like trace_id, request_id, or timestamps are more likely to hit this limit. Consider filtering by other labels first before exploring high-cardinality fields.

Showing partial results

If you see “Showing partial results for [field]” it means the query encountered an error but was still able to return some data. This commonly happens with timeout errors on large queries.

Try the following fixes:

Narrow your time range to make the query faster.
Add more filters to reduce the amount of data being processed.
If timeouts persist, contact your administrator to check Loki query timeout settings.

Pattern errors

This section covers error messages related to the Patterns feature.

There are no pattern matches

Pattern matching has not been configured.

Ensure pattern extraction is enabled by setting pattern_ingester.enabled=true in your Loki config. Learn about other necessary config.
Ensure the volume endpoint is enabled by setting volume_enabled=true within your Loki configuration file.

Note
The Patterns feature does not support multi-tenant or cross-stack data sources.

If you see a message in the UI on the Patterns tab that says “An error occurred within the plugin.” or receive an error message HTTP 500 with the message multiple org IDs present you have two options to work around the issue:

Use a single-stack Loki data source to restore Patterns.
Disable log patterns to continue using the other Logs Drilldown features with your multi-tenant or cross-stack data source. Go to Administration > Plugins and data > Plugins > Grafana Logs Drilldown and select the Disable patterns check box. This will disable use of the Patterns API and hide the Patterns tab in the user interface.

Sorry, we could not detect any patterns

This message appears when Loki’s pattern ingester is enabled but no patterns were extracted from your logs. This is rare but can happen with certain log formats.

Try the following fixes:

Check back later as patterns are continuously extracted from incoming logs.
Verify that your logs have enough volume for pattern detection to work effectively.
If the issue persists, reach out in the Grafana Labs community Slack channel or open an issue on GitHub.

Patterns are only available for the most recent 3 hours of data

Patterns in Loki are ephemeral and only stored for the most recent three hours. And patterns can change over time as your logging evolves. If your selected time range is entirely older than three hours ago, you won’t see any patterns.

To view patterns:

Adjust your time range to include some portion of the last three hours.
Use the time picker to select a more recent time window.

This pattern returns no logs

This error appears when viewing a pattern’s sample logs and the pattern query fails to return results. This can happen if the pattern was detected from logs that have since aged out or been deleted.

No action is typically needed. Try selecting a different pattern or adjusting your time range.

The logs returned by this pattern do not match the current query filters

This warning appears when you have active filters that exclude the logs matching the selected pattern. The pattern exists in your overall log data, but your current filters prevent those logs from appearing.

Try the following fixes:

Click the Clear filters button to remove conflicting filters.
Review your current label and line filters to understand why they exclude this pattern.
Adjust your filters to include the logs you want to see.

JSON panel issues

This section covers issues specific to the JSON visualization panel.

JSON filtering requires Loki 3.5.0

If you see “JSON filtering requires Loki 3.5.0. This view will be read only until Loki is upgraded to 3.5.0” it means your Loki instance doesn’t support the JSON filtering feature.

The JSON panel will still display your logs, but you won’t be able to filter by clicking on JSON field values.

To enable filtering:

Upgrade your Loki instance to version 3.5.0 or later.
Alternatively, switch to the Logs or Table visualization where filtering works with all Loki versions.

No JSON fields detected

This message appears when viewing the JSON panel but your log lines aren’t in JSON format.

This is informational rather than an error. If your logs aren’t JSON formatted:

Switch to the Logs or Table view for a better experience.
The JSON view is optimized for JSON-formatted log lines and may not display other formats correctly.

I cannot find something

Please open an issue on GitHub or get in touch privately and let us know what’s not working for you.

If you have something urgent, please get in touch via support. Grafana Cloud users can open a support ticket here.

Troubleshoot log queries (READ)

Thu, 09 Apr 2026 00:47:14 +0000

Troubleshoot log queries (READ)

This guide helps you troubleshoot errors that occur when querying logs from Loki. When Loki rejects or fails query requests, it’s typically due to query syntax errors, exceeding limits, timeout issues, or storage access problems.

Before you begin, ensure you have the following:

Access to Grafana Loki logs and metrics
Understanding of LogQL query language basics
Permissions to configure limits and settings if needed

Monitoring query errors

Query errors can be observed using these Prometheus metrics:

loki_request_duration_seconds - Query latency by route and status code
loki_logql_querystats_bytes_processed_per_seconds - Bytes processed during queries
loki_frontend_query_range_duration_seconds_bucket - Frontend query latency

You can set up alerts on 4xx and 5xx status codes to detect query problems early. This can be helpful when tuning limits configurations.

LogQL parse errors

Parse errors occur when the LogQL query syntax is invalid. Loki returns HTTP status code 400 Bad Request for all parse errors.

Error: Failed to parse the log query

Error message:

failed to parse the log query

Or with position details:

parse error at line <line>, col <col>: <message>

Cause:

The LogQL query contains syntax errors. This could be due to:

Missing or mismatched brackets, quotes, or braces
Invalid characters or operators
Incorrect function syntax
Invalid duration format

Common examples:

Invalid Query	Error	Fix
`{app="foo"`	Missing closing brace	`{app="foo"}`
`{app="foo"} \|= test`	Unquoted filter string	`{app="foo"} \|= "test"`
`rate({app="foo"}[5minutes])`	Invalid duration unit	`rate({app="foo"}[5m])`
`{app="foo"} \| json`	Missing pipe symbol before parser	`{app="foo"} \| json`

Resolution:

Start with a simple stream selector: {job="app"}, then add filters and operations incrementally to identify syntax issues.

Check bracket matching - Ensure all {, }, (, ), [, ] are properly closed.
Verify string quoting - All label values and filter strings must be quoted.
Use valid duration units - Use ns, us, ms, s, m, h, d, w, y , for example, 5m not 5minutes.
Review operator syntax - Ensure label matchers use proper operators (=, !=, =~, !~). Check the LogQL documentation for correct operator usage.
Use Grafana Assistant - If you are a Cloud Logs user, you can use Grafana Assistant to write or revise your query using natural language, for example, “What errors occurred for application foo in the last hour?”

Properties:

Enforced by: Query Frontend/Querier
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: At least one equality matcher required

Error message:

parse error : queries require at least one regexp or equality matcher that does not have an empty-compatible value. For instance, app=~".*" does not meet this requirement, but app=~".+" will

Cause:

The query uses only negative matchers (!=, !~) or matchers that match empty strings (=~".*"), which would select all streams. This is prevented to protect against accidentally querying the entire database.

Invalid examples:

{foo!="bar"}
{app=~".*"}
{foo!~"bar|baz"}

Valid examples:

{foo="bar"}
{app=~".+"}
{app="baz", foo!="bar"}

Resolution:

Add at least one positive matcher that selects specific streams.
Use .+ instead of .* in regex matchers to require at least one character.
Add additional label selectors to narrow down the query scope.
Use Grafana Assistant - If you are a Cloud Logs user, you can use Grafana Assistant to write or revise your query using natural language, for example, “Find logs containing ’foo’ but not ‘bar’ or ‘baz’.”

Properties:

Enforced by: Query Frontend
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Only label matchers are supported

Error message:

only label matchers are supported

Cause:

The query was passed to an API that only accepts label matchers (like the series API), but included additional expressions like line filters or parsers.

Resolution:

Use only stream selectors for APIs that don’t support full LogQL:

# Valid for series API
{app="foo", env="prod"}

# Invalid for series API
{app="foo"} |= "error"

Properties:

Enforced by: API handler
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Log queries not supported as instant query type

Error message:

log queries are not supported as an instant query type, please change your query to a range query type

Cause:

A log query (one that returns log lines rather than metrics) was submitted to the instant query endpoint (/loki/api/v1/query). Log queries must use the range query endpoint.

Resolution:

Convert to a range query Convert log queries to range queries with a time range. Range queries are the default in Grafana Explore.
Use the range query endpoint /loki/api/v1/query_range for log queries.

Convert to a metric query if you need to use instant queries:

# This is a log query (returns logs)
{app="foo"} |= "error"

# This is a metric query (can be instant)
count_over_time({app="foo"} |= "error"[5m])

Use Grafana Assistant - If you are a Cloud Logs user, you can use Grafana Assistant to write or revise your query.

Properties:

Enforced by: Query API
Retryable: No (use correct endpoint or query type)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Invalid aggregation without unwrap

Error message:

parse error : invalid aggregation sum_over_time without unwrap

Cause:

Aggregation functions like sum_over_time, avg_over_time, min_over_time, max_over_time require an unwrap expression to extract a numeric value from log lines.

Resolution:

Add an unwrap expression to extract the numeric label:

# Invalid
sum_over_time({app="foo"} | json [5m])

# Valid - unwrap a numeric label
sum_over_time({app="foo"} | json | unwrap duration [5m])

Properties:

Enforced by: Query Parser
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Invalid aggregation with unwrap

Error message:

parse error : invalid aggregation count_over_time with unwrap

Cause:

The count_over_time function doesn’t use unwrapped values - it just counts log lines. Using it with unwrap is invalid.

Resolution:

Remove the unwrap expression for count_over_time:

# Invalid
count_over_time({app="foo"} | json | unwrap duration [5m])

# Valid
count_over_time({app="foo"} | json [5m])

Use sum_over_time if you want to sum unwrapped values.

Properties:

Enforced by: Query Parser
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Query limit errors

These errors occur when queries exceed configured resource limits. They return HTTP status code 400 Bad Request.

Error: Maximum series reached

Error message:

maximum number of series (<limit>) reached for a single query; consider reducing query cardinality by adding more specific stream selectors, reducing the time range, or aggregating results with functions like sum(), count() or topk()

Cause:

The query matches more unique label combinations (series) than the configured limit allows. This protects against queries that would consume excessive memory.

Default configuration:

max_query_series: 500 (default)

Resolution:

Add more specific stream selectors to reduce cardinality:

# Too broad
{job="ingress-nginx"}

# More specific
{job="ingress-nginx", namespace="production", pod=~"ingress-nginx-.*"}

Reduce the time range of the query.
Use label filters to narrow down results: {job="app"} |= "error"
Use aggregation functions to reduce cardinality:
logql
```
sum by (status) (rate({job="nginx"} | json [5m]))
```

Increase the limit if resources allow:

limits_config:
  max_query_series: 1000  #default is 500

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Cardinality issues

Error message:

cardinality limit exceeded for {}; 100001 entries, more than limit of 100000

Cause:

The query produces results with too many unique label combinations. This protects against queries that would generate excessive memory usage and slow performance.

Default configuration:

cardinality_limit: 100000

Resolution:

Use more specific label selectors to reduce the number of unique streams.
Apply aggregation functions to reduce cardinality:
logql
```
sum by (status) (rate({job="nginx"}[5m]))
```

Use by() or without() clauses to group results and reduce dimensions:

sum by (status, method) (rate({job="nginx"} | json [5m]))

Another alternative is using drop or keep to reduce the number of labels and hence the cardinality:

# Drop high-cardinality labels like request_id or trace_id
{job="nginx"} | json | drop request_id, trace_id, session_id

# Keep only the labels you need
{job="nginx"} | json | keep status, method, path

Increase the limit if needed:

limits_config:
  cardinality_limit: 200000  #default is 100000

Properties:

Enforced by: Query Engine
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Max entries limit per query exceeded

Error message:

max entries limit per query exceeded, limit > max_entries_limit_per_query (<requested> > <limit>)

Cause:

The query requests more log entries than the configured maximum. This applies to log queries (not metric queries).

Default configuration:

max_entries_limit_per_query: 5000

Resolution:

Reduce the limit parameter in your query request.
Add more specific filters to return fewer results:
logql
```
{app="foo"} |= "error" 
```
Reduce the time range of the query.

Increase the limit if needed:

limits_config:
  max_entries_limit_per_query: 10000  #default is 5000

Properties:

Enforced by: Querier/Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Query would read too many bytes

Error message:

the query would read too many bytes (query: <size>, limit: <limit>); consider adding more specific stream selectors or reduce the time range of the query

Cause:

The estimated data volume for the query exceeds the configured limit. This is determined before query execution using index statistics.

Default configuration:

max_query_bytes_read: 0B (disabled by default)

Resolution:

Add more specific stream selectors to reduce data volume.
Reduce the time range of the query.
Increase the limit if resources allow:
YAML
```
limits_config:
  max_query_bytes_read: 10GB
```

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Too many chunks (count)

Error message:

the query hit the max number of chunks limit (limit: 2000000 chunks)

Cause:

The number of chunks that the query would read exceeds the configured limit. This protects against queries that would scan excessive amounts of data and consume too much memory.

Default configuration:

max_chunks_per_query: 2000000

Resolution:

Narrow stream selectors to reduce the number of matching chunks:

# Too broad
{job="app"}

# More specific
{job="app", environment="production", namespace="api"}

Reduce the query time range to scan fewer chunks.

Increase the limit if resources allow:

limits_config:
  max_chunks_per_query: 5000000  #default is 2000000

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Stream matcher limits

Error message:

max streams matchers per query exceeded, matchers-count > limit (1500 > 1000)

Cause:

The query contains too many stream matchers. This limit prevents queries with excessive complexity that could impact query performance.

Default configuration:

max_streams_matchers_per_query: 1000

Resolution:

Simplify your query by using fewer label matchers.
Combine multiple queries instead of using many OR conditions.

Use regex matchers to consolidate multiple values:

# Good: 3 matchers using regex patterns
{cluster="prod", namespace=~"api|web", pod=~"nginx-.*"}

Increase the limit if needed:

limits_config:
  max_streams_matchers_per_query: 2000  #default is 1000

Properties:

Enforced by: Querier
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Query too large for single querier

Error message:

query too large to execute on a single querier: (query: <size>, limit: <limit>); consider adding more specific stream selectors, reduce the time range of the query, or adjust parallelization settings

Or for un-shardable queries:

un-shardable query too large to execute on a single querier: (query: <size>, limit: <limit>); consider adding more specific stream selectors or reduce the time range of the query

Cause:

Even after query splitting and sharding, individual query shards exceed the per-querier byte limit.

Default configuration:

max_querier_bytes_read: 150GB (per querier)

Resolution:

Add more specific stream selectors.
Reduce the time range or Break large queries into smaller time ranges.
Simplify the query if possible - some queries cannot be sharded.

Increase the limit (requires more querier resources):

limits_config:
  max_querier_bytes_read: 200GB  # default is 150GB

Scale querier resources

Properties:

Enforced by: Query Frontend
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Interval value exceeds limit

Error message:

[interval] value exceeds limit

Cause:

The range vector interval (in brackets like [5m]) exceeds configured limits.

Resolution:

Reduce the range interval in your query:

# If [1d] is too large, try smaller intervals
rate({app="foo"}[1h])

Check your configuration for max_query_length limits. The default is 30d1h.

Properties:

Enforced by: Query Engine
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Time range errors

These errors relate to the time range specified in queries.

Error: Query time range exceeds limit

Error message:

the query time range exceeds the limit (query length: <duration>, limit: <limit>)

Cause:

The difference between the query’s start and end time exceeds the maximum allowed query length.

Default configuration:

max_query_length: 721h (30 days + 1 hour)

Resolution:

Reduce the query time range:

# Instead of querying 60 days
logcli query '{app="foo"}' --from="60d" --to="now"

# Query 30 days or less
logcli query '{app="foo"}' --from="30d" --to="now"

Increase the limit if storage retention supports it:
YAML
```
limits_config:
  max_query_length: 2160h  # 90 days
```

Properties:

Enforced by: Query Frontend/Querier
Retryable: No (query must be modified)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Data is no longer available

Error message:

this data is no longer available, it is past now - max_query_lookback (<duration>)

Cause:

The entire query time range falls before the max_query_lookback limit. This happens when trying to query data older than the configured lookback period.

Default configuration:

max_query_lookback: 0 (The default value of 0 does not set a limit.)

Resolution:

Query more recent data within the lookback window.
Adjust the lookback limit if the data should be queryable:
YAML
```
limits_config:
  max_query_lookback: 8760h  # 1 year
```
Caution
The lookback limit should not exceed your retention period.

Properties:

Enforced by: Query Frontend/Querier
Retryable: No
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Invalid query time range

Error message:

invalid query, through < from (<end> < <start>)

Cause:

The query end time is before the start time, which is invalid.

Resolution:

Swap start and end times if they were reversed.
Check timestamp formats to ensure times are correctly specified.

Properties:

Enforced by: Query Frontend/Querier
Retryable: No (query must be fixed)
HTTP status: 400 Bad Request
Configurable per tenant: No

Required labels errors

These errors occur when queries don’t meet configured label requirements.

Error: Missing required matchers

Error message:

stream selector is missing required matchers [<required_labels>], labels present in the query were [<present_labels>]

Cause:

The tenant is configured to require certain label matchers in all queries, but the query doesn’t include them.

Default configuration:

required_labels: [] (none required by default)

Resolution:

Check with your administrator about which labels are required.

Add the required labels to your query:

# If 'namespace' is required
{app="foo", namespace="production"}

Properties:

Enforced by: Query Frontend
Retryable: No (query must include required labels)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Not enough label matchers

Error message:

stream selector has less label matchers than required: (present: [<labels>], number_present: <count>, required_number_label_matchers: <required>)

Cause:

The tenant is configured to require a minimum number of label matchers, but the query has fewer.

Default configuration:

minimum_labels_number: 0 (no minimum by default)

Resolution:

Add more label matchers to meet the minimum requirement:

# If minimum is 2, add another selector
{app="foo", namespace="production"}

Properties:

Enforced by: Query Frontend
Retryable: No (query must meet requirements)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Timeout errors

Timeout errors occur when queries take too long to execute.

Error: Request timed out

Error message:

request timed out, decrease the duration of the request or add more label matchers (prefer exact match over regex match) to reduce the amount of data processed

Or:

context deadline exceeded

Cause:

The query exceeded the configured timeout. This can happen due to:

Large time ranges
High cardinality queries
Complex query expressions
Insufficient cluster resources
Network issues

Default configuration:

query_timeout: 1m
server.http_server_read_timeout: 30s
server.http_server_write_timeout: 30s

Resolution:

Reduce the time range of the query.

Add more specific filters to reduce data processing:

# Less specific (slower)
{namespace=~"prod.*"}

# More specific (faster)
{namespace="production"}

Prefer exact matchers over regex when possible.
Add line filters early in the pipeline:
logql
```
{app="foo"} |= "error" | json | level="error"
```

Increase timeout limits (if resources allow):

limits_config:
  query_timeout: 5m

server:
  http_server_read_timeout: 5m
  http_server_write_timeout: 5m

Use sampling for exploratory queries

 {job="app"} | line_format "{{__timestamp__}} {{.msg}}" | sample 0.1

Check for network issues between components.

Properties:

Enforced by: Query Frontend/Querier
Retryable: Yes (with modifications)
HTTP status: 504 Gateway Timeout
Configurable per tenant: Yes (query_timeout)

Error: Request cancelled by client

Error message:

the request was cancelled by the client

Cause:

The client closed the connection before receiving a response. This is typically caused by:

Client-side timeout
User navigating away in Grafana
Network interruption

Resolution:

Increase client timeout in Grafana or LogCLI.
Optimize the query to return faster.
Check network connectivity between client and Loki.

Properties:

Enforced by: Client
Retryable: Yes
HTTP status: 499 Client Closed Request
Configurable per tenant: No

Query blocked errors

These errors occur when queries are administratively blocked.

Error: Query blocked by policy

Error message:

query blocked by policy

Cause:

The query matches a configured block rule. Administrators create tenant policies and rate limiting rules to block specific queries or query patterns to protect the cluster from expensive or problematic queries.

Resolution:

Check with your Loki administrator about blocked queries and to review policy settings.
Modify the query to avoid the block pattern:
- Change the stream selectors
- Adjust the time range
- Use different aggregations
Request the block to be removed if the query is legitimate.

Configuration reference:

limits_config:
  blocked_queries:
    - pattern: ".*"          # Regex pattern to match
      regex: true
      types:                 # Query types to block
        - metric
        - filter
      hash: 0               # Or block specific query hash

Properties:

Enforced by: Query Engine
Retryable: No (unless block is removed)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Querying is disabled

Error message:

querying is disabled, please contact your Loki operator

Cause:

Query parallelism is set to 0, effectively disabling queries for the tenant.

Resolution:

Contact your Loki administrator to enable querying.

Check configuration for max_query_parallelism:

limits_config:
  max_query_parallelism: 32     #(the default)

Properties:

Enforced by: Query Frontend
Retryable: No (until configuration is fixed)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Error: Multi variant queries disabled

Multi variant queries are an experimental feature that enables support for running multiple query variants over the same underlying data. For example, running both a rate() and count_over_time() query over the same range selector.

Error message:

multi variant queries are disabled for this instance

Cause:

The query uses the variants feature, but it’s disabled for the tenant or instance.

Resolution:

Remove variant expressions from the query.

Enable the feature if needed:

limits_config:
  enable_multi_variant_queries: true  #default is false

Properties:

Enforced by: Query Engine
Retryable: No (until feature is enabled)
HTTP status: 400 Bad Request
Configurable per tenant: Yes

Pipeline processing errors

Pipeline errors occur during log line processing but don’t cause query failures. Instead, affected log lines are annotated with error labels.

Understanding pipeline errors

When a pipeline stage fails (for example, parsing JSON that isn’t valid JSON), Loki:

Does NOT filter out the log line
Adds an __error__ label with the error type
Optionally adds __error_details__ with more information
Passes the log line to the next pipeline stage

Error types

Error Label Value	Cause
`JSONParserErr`	Log line is not valid JSON
`LogfmtParserErr`	Log line is not valid logfmt
`SampleExtractionErr`	Failed to extract numeric value for metrics
`LabelFilterErr`	Label filter operation failed
`TemplateFormatErr`	Template formatting failed

Viewing pipeline errors

To see logs with errors:

{app="foo"} | json | __error__!=""

To see error details:

{app="foo"} | json | __error__!="" | line_format "Error: {{.__error__}} - {{.__error_details__}}"

Filtering out errors

To exclude logs with parsing errors:

{app="foo"} | json | __error__=""

To exclude specific error types:

{app="foo"} | json | __error__!="JSONParserErr"

Dropping error labels

To remove error labels from results:

{app="foo"} | json | drop __error__, __error_details__

Authentication and connection errors

These errors occur when connecting to Loki, often when using LogCLI.

Error: No org ID

Error message:

no org id

Cause:

Multi-tenancy is enabled but no tenant ID was provided in the request.

Resolution:

Add the X-Scope-OrgID header in your request.

For LogCLI, use the --org-id flag:

logcli query '{app="foo"}' --org-id="my-tenant"

In Grafana, configure the tenant ID in the data source settings.

Properties:

Enforced by: Loki API
Retryable: Yes (with tenant ID)
HTTP status: 400 Bad Request
Configurable per tenant: No

Error: Authentication configuration conflict

Error message:

at most one of HTTP basic auth (username/password), bearer-token & bearer-token-file is allowed to be configured

Or:

at most one of the options bearer-token & bearer-token-file is allowed to be configured

Cause:

Multiple authentication methods are configured simultaneously in LogCLI.

Resolution:

Use only one authentication method:

# Basic auth
logcli query '{app="foo"}' --username="user" --password="pass"

# OR bearer token
logcli query '{app="foo"}' --bearer-token="token"

# OR bearer token file
logcli query '{app="foo"}' --bearer-token-file="/path/to/token"

Properties:

Enforced by: LogCLI
Retryable: Yes (with correct configuration)
HTTP status: N/A (client-side error)
Configurable per tenant: No

Error: Run out of attempts while querying

Error message:

run out of attempts while querying the server

Cause:

LogCLI exhausted all retry attempts when trying to reach Loki. This usually indicates:

Network connectivity issues
Server unavailability
Authentication failures

Resolution:

Check Loki server availability.
Verify network connectivity.
Check authentication credentials.
Increase retries if transient issues are expected:
Bash
```
logcli query '{app="foo"}' --retries=5
```

Properties:

Enforced by: LogCLI
Retryable: Yes (automatic retries exhausted)
HTTP status: Varies
Configurable per tenant: No

Error: WebSocket connection closed unexpectedly

Error message:

websocket: close 1006 (abnormal closure): unexpected EOF

Cause:

When tailing logs, the WebSocket connection was closed unexpectedly. This can happen if:

The querier handling the tail request stopped
Network interruption occurred
Server-side timeout

Resolution:

LogCLI will automatically attempt to reconnect, up to 5 times.
Check Loki querier health if reconnections fail.
Review network stability between client and server.

Properties:

Enforced by: Network/Server
Retryable: Yes (automatic reconnection)
HTTP status: N/A (WebSocket error)
Configurable per tenant: No

Data availability errors

These errors occur when requested data is not available.

Error: No data found

Error message:

no data found

Or an empty result set with no error message.

Cause:

The query time range contains no matching log data. This can happen if:

No logs match the stream selectors
The time range is outside the data retention period
Log ingestion is not working
Stream labels don’t match any existing streams

Resolution:

Verify the time range contains data for your streams.

Check if log ingestion is working correctly:

# Check if any data is being ingested
logcli query '{job=~".+"}'

Verify stream selectors match existing log streams:

# List available streams
curl http://loki:3100/loki/api/v1/series

Check data retention settings to ensure logs are still available.
Use broader selectors to test if any data exists:
logql
```
{job=~".+"}
```

Properties:

Enforced by: Query Engine
Retryable: Yes (with different parameters)
HTTP status: 200 OK (with empty result)
Configurable per tenant: No

Error: Index not ready

Error message:

index not ready

Or:

index gateway not ready for time range

Cause:

The index for the requested time range is not yet available for querying. This can happen when:

Index files are still being synced from storage
The index gateway is still starting up
Querying data older than the configured ready index period

Default configuration:

query_ready_index_num_days: 0 (all indexes are considered ready)

Resolution:

Wait for the index to become available - this is often a temporary issue during startup.
Query more recent data that’s available in ingesters:
logql
```
{app="foo"} # Query last few hours instead of older data
```

Check the configuration for index readiness:

query_range:
  query_ready_index_num_days: 7  #default is 0

Verify index synchronization is working correctly by checking ingester and index gateway logs.

Properties:

Enforced by: Index Gateway/Querier
Retryable: Yes (wait and retry)
HTTP status: 503 Service Unavailable
Configurable per tenant: No

Error: Tenant limits

Error message:

max concurrent tail requests limit exceeded, count > limit (10 > 5)

Cause:

The tenant has exceeded the maximum number of concurrent streaming (tail) requests. This limit protects the cluster from excessive resource consumption by real-time log streaming.

Default configuration:

max_concurrent_tail_requests: 10

Resolution:

Reduce the number of concurrent tail/streaming queries.

Use batch queries instead of real-time streaming where possible:

# Instead of tailing in real-time
# Use periodic range queries
{app="foo"} |= "error"

Increase the limit if more concurrent tails are needed:

limits_config:
  max_concurrent_tail_requests: 20  #default is 10

Properties:

Enforced by: Querier
Retryable: Yes (when connections are available)
HTTP status: 429 Too Many Requests
Configurable per tenant: Yes

Storage errors

These errors occur when Loki cannot read data from storage.

Error: Failed to load chunk

Error message:

failed to load chunk '<chunk_key>'

Cause:

Loki couldn’t retrieve a chunk from object storage. Possible causes:

Chunk was deleted or moved
Storage permissions issue
Network connectivity to storage
Storage service unavailable

Resolution:

Check storage connectivity from Loki components.
Verify storage credentials and permissions.
Check for chunk corruption or deletion.
Review storage service status.

Properties:

Enforced by: Storage Client
Retryable: Yes (automatically)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Error: Object not found in storage

Error message:

object not found in storage

Cause:

The requested chunk or object doesn’t exist in storage. This might happen if:

Data was deleted due to retention
Compaction removed the chunk
Chunk was never written successfully

Resolution:

Check if data is within retention period.
Verify data was ingested successfully.
Review compaction jobs for issues.

Properties:

Enforced by: Storage Client
Retryable: No (data doesn’t exist)
HTTP status: 404 or 500 depending on context
Configurable per tenant: No

Error: Failed to decode chunk

Error message:

failed to decode chunk '<chunk_key>' for tenant '<tenant>': <error>

Cause:

A chunk was retrieved from storage but couldn’t be decoded. This indicates chunk corruption.

Resolution:

Report to Loki administrators for investigation.
Check for storage data integrity issues.
Note that the corrupted chunk data may be unrecoverable.

Properties:

Enforced by: Storage Client
Retryable: No (chunk is corrupted)
HTTP status: 500 Internal Server Error
Configurable per tenant: No

Troubleshooting workflow

Follow this workflow when investigating query issues:

Check the error message - Identify which category of error you’re encountering.
Review query syntax - Use the LogQL documentation to validate your query.
Check query statistics - In Grafana, enable “Query Inspector” to see:
- Bytes processed
- Number of chunks scanned
- Execution time breakdown

Simplify the query - Start with a basic selector and add complexity:

# Start simple
{app="foo"}

# Add filters
{app="foo"} |= "error"

# Add parsing
{app="foo"} |= "error" | json

# Add label filters
{app="foo"} |= "error" | json | level="error"

Check metrics for query performance:

# Query latency
histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, route))

# Query errors
sum by (status_code) (rate(loki_request_duration_seconds_count[5m]))

Review Loki logs for detailed error information:

kubectl logs -l app=loki-read --tail=100 | grep -i error

Test with LogCLI for more detailed output:
Bash
```
logcli query '{app="foo"}' --stats --limit=10
```

Learn more about LogQL Query Language
Configure appropriate query limits
Learn more about Query performance tuning
Review the LogCLI documentation
Learn more about LogQL query optimization
Review query performance best practices
Use query debugging features to analyze slow queries
Explore the Grafana Loki GitHub repository for community support