This is documentation for the next version of Grafana Loki documentation. For the latest stable release, go to the latest version.

Documentation Grafana Loki Manage Troubleshooting Troubleshoot ingestion

Troubleshoot log ingestion (WRITE)

This guide helps you troubleshoot errors that occur when ingesting logs into Loki and writing logs to storage. When Loki rejects log ingestion requests, it’s typically due to exceeding rate limits, violating validation rules, or encountering storage issues.

Before you begin, ensure you have the following:

• Access to Grafana Loki logs and metrics
• Understanding of your log ingestion pipeline (Alloy or custom clients)
• Permissions to configure limits and settings if needed

Monitoring ingestion errors

All ingestion errors are tracked using Prometheus metrics:

• loki_discarded_samples_total - Count of discarded log samples by reason
• loki_discarded_bytes_total - Volume of discarded log data by reason

The reason label on these metrics indicates the specific error type. Set up alerts on these metrics to detect ingestion problems.

Rate limit errors

Rate limits protect Loki from being overwhelmed by excessive log volume. When rate limits are exceeded, Loki returns HTTP status code 429 Too Many Requests.

Error: rate_limited

Error message:

ingestion rate limit exceeded for user <tenant> (limit: <limit> bytes/sec) while attempting to ingest '<lines>' lines totaling '<bytes>' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased

If a per-policy ingestion rate override applies to the stream (configured with ingestion_rate_mb/ingestion_burst_size_mb under a policy in policy_stream_mapping), the message instead identifies the policy:

ingestion rate limit exceeded for user <tenant>, policy <policy> (limit: <limit> bytes/sec) while attempting to ingest '<lines>' lines totaling '<bytes>' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased

If a single request exceeds more than one rate-limit bucket at once (for example the tenant default and a specific policy), Loki returns a single combined error instead, enumerating each exceeded bucket:

ingestion rate limit exceeded for user <tenant> across multiple buckets: the tenant default (limit: <limit> bytes/sec) ingesting <lines> lines totaling <bytes> bytes; policy "<policy>" (limit: <limit> bytes/sec) ingesting <lines> lines totaling <bytes> bytes; reduce log volume or contact your Loki administrator to see if the limits can be increased

In all three variants, the reason label reported in loki_discarded_samples_total/loki_discarded_bytes_total is still rate_limited; there’s no separate reason for the policy-scoped or multi-bucket cases.

Cause:

The tenant has exceeded their configured ingestion rate limit. This is a global per-tenant limit enforced by the distributor. If ingestion_rate_mb or ingestion_burst_size_mb is overridden for a specific ingestion policy, that policy’s limit is checked separately from the tenant default, which is why the error message can reference a policy or multiple buckets.

Default configuration:

• ingestion_rate_mb: 4 MB/sec
• ingestion_burst_size_mb: 6 MB
• ingestion_rate_strategy: “global” (shared across all distributors)

Resolution:

• Increase rate limits (if you have sufficient cluster resources):

  YAML Copy

  limits_config:
    ingestion_rate_mb: 8
    ingestion_burst_size_mb: 12

• Reduce log volume by:

  • Filtering unnecessary log lines using Alloy processing stages
  • Collecting logs from fewer targets
  • Sampling high-volume streams
  • Using Alloy’s rate limiting and filtering capabilities

• Switch to local strategy (if using single-region deployment):

  YAML Copy

  limits_config:
    ingestion_rate_strategy: local

  Note

  This multiplies the effective limit by the number of distributor replicas.

Properties:

• Enforced by: Distributor
• Retryable: Yes
• Sample discarded: No (the request is rejected before Loki accepts any of its samples, though loki_discarded_samples_total and loki_discarded_bytes_total are still incremented)
• HTTP status: 429 Too Many Requests
• Configurable per tenant: Yes

Error: per_stream_rate_limit

Error message:

Per stream rate limit exceeded (limit: <limit>/sec) while attempting to ingest for stream '<stream_labels>' totaling <bytes>, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased

Cause:

A single stream has exceeded its own rate limit. Each stream has a rate limit applied to it to prevent individual streams from overwhelming the set of ingesters it is distributed to. This limit is enforced by the ingester, not the distributor.

Default configuration:

• per_stream_rate_limit: 3 MB/sec
• per_stream_rate_limit_burst: 15 MB

Resolution:

• Split the stream into several smaller streams by adding more distinguishing labels, so the write volume is spread across more streams.

• Reduce the rate of writes to the stream, for example with an Alloy stage.limit block.

• Increase the per-stream limits (we typically recommend no higher than 5MB for per_stream_rate_limit and 20MB for per_stream_rate_limit_burst):

  YAML Copy

  limits_config:
    per_stream_rate_limit: 5MB
    per_stream_rate_limit_burst: 20MB

Properties:

• Enforced by: Ingester
• Retryable: Yes
• Sample discarded: No
• HTTP status: 429 Too Many Requests
• Configurable per tenant: Yes

Error: stream_limit

Error message:

maximum active stream limit exceeded when trying to create stream <stream_labels>, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: '<tenant>'

Cause:

The tenant has reached the maximum number of active streams. Active streams are held in memory on ingesters, and excessive streams can cause out-of-memory errors.

Default configuration:

• max_global_streams_per_user: 5000 (globally)
• max_streams_per_user: 0 (no local limit per ingester)
• Active stream window: chunk_idle_period (default: 30 minutes)

Terminology:

• Active stream: A stream that has received logs within the chunk_idle_period (default: 30 minutes)

Resolution:

• Reduce stream cardinality by:

  • Using fewer unique label combinations by removing unnecessary labels
  • Avoiding high-cardinality labels (IDs, timestamps, UUIDs)
  • Using structured metadata instead of labels for high-cardinality data

• Increase stream limits (if you have sufficient memory):

  YAML Copy

  limits_config:
    max_global_streams_per_user: 10000

• Check for a recent ingester rollout or scaling event before increasing limits. If stream_limit errors started when ingesters were added, restarted, or migrated, the tenant may not have a real cardinality problem. Increasing the limit will not prevent this transient effect and can mask genuine cardinality growth.

Properties:

• Enforced by: Ingester
• Retryable: Yes
• Sample discarded: No
• HTTP status: 429 Too Many Requests
• Configurable per tenant: Yes

Validation errors

Validation errors occur when log data doesn’t meet Loki’s requirements. These return HTTP status code 400 Bad Request and are not retryable.

Error: Missing streams

Error message:

error at least one valid stream is required for ingestion

Cause:

The push request did not contain any valid streams at all (an empty streams list). This is checked before any per-stream validation runs, so it is distinct from missing_labels, which applies when a stream is present but has no labels.

Default configuration:

This validation is always enabled and cannot be disabled.

Resolution:

• Check your log shipping client configuration to make sure push requests always include at least one stream.
• Check for empty batches if your pipeline batches log lines before sending; an empty batch should not trigger a push request.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: No (the whole request is rejected before any streams can be counted; this reason is not tracked in loki_discarded_samples_total or loki_discarded_bytes_total)
• HTTP status: 422 Unprocessable Entity
• Configurable per tenant: No

Error: line_too_long

Error message:

max entry size '<max_size>' bytes exceeded for stream '<stream_labels>' while adding an entry with length '<entry_size>' bytes

Cause:

A log line exceeds the maximum allowed size.

Default configuration:

• max_line_size: 256 KB
• max_line_size_truncate: false
• max_line_size_truncate_identifier: "" (no marker appended)

Resolution:

• Truncate long lines instead of discarding them:

  YAML Copy

  limits_config:
    max_line_size: 256KB
    max_line_size_truncate: true

  Note

  When max_line_size_truncate is enabled, truncated lines are still ingested, so they are counted in loki_mutated_samples_total and loki_mutated_bytes_total instead of loki_discarded_samples_total. Set max_line_size_truncate_identifier to append a marker string (for example "[truncated]") to the end of each truncated line, so truncated lines are easy to identify later when you’re querying.

• Increase the line size limit (not recommended above 256KB):

  YAML Copy

  limits_config:
    max_line_size: 256KB

  Warning

  Loki was built as a large multi-user, multi-tenant database and as such this limit becomes very important to maintain stability and performance with many users query the database simultaneously. We strongly recommend against increasing the max_line_size, doing so will make it very difficult to provide consistent query performance and stability of the system without having to throw extremely large amounts of memory and/or increasing the GRPC message size limits to really high levels, both of which will likely lead to poorer performance and worse experiences.

• Filter or truncate logs before sending using Alloy processing stages:

  Alloy Copy

  stage.replace {
    expression = "^(.{10000}).*$"
    replace    = "$1"
  }

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes, unless max_line_size_truncate is enabled, in which case the entry is truncated and ingested
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: invalid_labels

Error message:

error parsing labels '<labels>' with error: <parse_error>

Cause:

Label names or values contain invalid characters or don’t follow Prometheus naming conventions:

• Label names must match regex: [a-zA-Z_][a-zA-Z0-9_]*
• Label names cannot start with __ (reserved for internal use)
• Labels must be valid UTF-8
• Properly escape special characters in label values
• Ensure label syntax follows PromQL format: {key="value"}

Default configuration:

This validation is always enabled and cannot be disabled.

Resolution:

• Fix label names in your log shipping configuration:

  YAML Copy

  # Incorrect
  labels:
    "123-app": "value"      # Starts with number
    "app-name": "value"     # Contains hyphen
    "__internal": "value"   # Reserved prefix
  
  # Correct
  labels:
    app_123: "value"
    app_name: "value"
    internal_label: "value"

• Use Alloy’s relabeling stages to fix labels:

  Alloy Copy

  stage.label_drop {
    values = ["invalid_label"]
  }
  
  stage.labels {
    values = {
      app_name = "",
    }
  }

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: No

Error: missing_labels

Error message:

error at least one label pair is required per stream

Cause:

A log stream was submitted without any labels. Loki requires at least one label pair for each log stream.

Default configuration:

This validation is always enabled and cannot be disabled. Every stream must have at least one label.

Resolution:

Ensure all log streams have at least one label. Update your log shipping configuration to include labels:

Alloy Copy

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.source.file "logs" {
  targets = [
    {
      __path__ = "/var/log/*.log",
      job      = "myapp",
      environment = "production",
    },
  ]
  forward_to = [loki.write.default.receiver]
}

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: No

Error: too_far_behind

These errors occur when log entries arrive in the wrong chronological order.

Error messages:

• entry too far behind, entry timestamp is: <timestamp>, oldest acceptable timestamp is: <cutoff>

Cause:

Logs are being ingested with timestamps that violate Loki’s ordering constraints:

• Logs older than half of max_chunk_age from the newest entry are rejected

Default configuration:

• Loki always accepts out-of-order writes; the per-tenant unordered_writes setting and the -ingester.unordered-writes flag that used to control this have been removed.
• max_chunk_age: 2 hours
• Acceptable timestamp window: 1 hour behind the newest entry (half of max_chunk_age)

Resolution:

• Ensure timestamps are assigned correctly:

  • Use a timestamp parsing stage in Alloy if logs have embedded timestamps
  • Avoid letting Alloy assign timestamps at read time for delayed log delivery
  Alloy Copy

  stage.timestamp {
    source = "timestamp_field"
    format = "RFC3339"
  }

• Check for clock skew between log sources and Loki. Ensure clocks are synchronized across your infrastructure.

Properties:

• Enforced by: Ingester
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: No, max_chunk_age is global only.

Error: greater_than_max_sample_age

Error message:

entry for stream '<stream_labels>' has timestamp too old: <timestamp>, oldest acceptable timestamp is: <cutoff>

Cause:

The log entry’s timestamp is older than the configured maximum sample age. This prevents ingestion of very old logs.

Default configuration:

• reject_old_samples: true
• reject_old_samples_max_age: 168 hours (7 days)

Resolution:

• Increase the maximum sample age:

  YAML Copy

  limits_config:
    reject_old_samples_max_age: 336h  # 14 days

• Fix log delivery delays causing old timestamps.

• For historical log imports, temporarily disable the check per tenant.

• Check for clock skew between log sources and Loki. Ensure clocks are synchronized across your infrastructure.

• Disable old sample rejection (not recommended):

  YAML Copy

  limits_config:
    reject_old_samples: false

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: too_far_in_future

Error message:

entry for stream '<stream_labels>' has timestamp too new: <timestamp>

Cause:

The log entry’s timestamp is further in the future than the configured grace period allows.

Default configuration:

• creation_grace_period: 10 minutes

Resolution:

• Check for clock skew between log sources and Loki. Ensure clocks are synchronized across your infrastructure.

• Verify application timestamps. Validate timestamp generation in your applications.

• Verify timestamp parsing in your log shipping configuration.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: max_label_names_per_series

Error message:

entry for stream '<stream_labels>' has <count> label names; limit <limit>

Cause:

The stream has more labels than allowed.

Default configuration:

• max_label_names_per_series: 15

Resolution:

• Reduce the number of labels by:
  • Using structured metadata for high-cardinality data
  • Removing unnecessary labels
  • Combining related labels

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: label_name_too_long

Error message:

stream '<stream_labels>' has label name too long: '<label_name>'

Cause:

A label name exceeds the maximum allowed length.

Default configuration:

• max_label_name_length: 1024 bytes

Resolution:

• Shorten label names in your log shipping configuration to under the configured limit. You can use abbreviations or shorter descriptive names.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: label_value_too_long

Error message:

stream '<stream_labels>' has label value too long: '<label_value>'

Cause:

A label value exceeds the maximum allowed length.

Default configuration:

• max_label_value_length: 2048 bytes

Resolution:

• Shorten label values in your log shipping configuration to under the configured limit. You can use hash values for very long identifiers.

• Use structured metadata for long values instead of labels.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: duplicate_label_names

Error message:

stream '<stream_labels>' has duplicate label name: '<label_name>'

Cause:

The stream has two or more labels with identical names.

Default configuration:

This validation is always enabled and cannot be disabled. Duplicate label names are never allowed.

Resolution:

• Remove duplicates. Remove duplicate label definitions in your log shipping configuration to ensure unique label names per stream.

• Check clients. Check your ingestion pipeline for label conflicts.

• Verify processing. Verify your label processing and transformation rules.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: No

Error: request_body_too_large

Error message:

request body too large: <size> bytes, limit: <limit> bytes

Cause:

This error message can come from three different places:

• A gateway/reverse proxy in front of Loki enforcing its own body-size limit.
• Loki’s distributor rejecting a push request that exceeds its own configured limit (limits_config.max_push_size) before it can parse the streams out of the body.
• A single stream’s label string exceeding 16MB. This limit is hardcoded and cannot be changed through any configuration option; it exists because the underlying label parser cannot handle labels above this size.

Default configuration:

• Loki distributor push size: limits_config.max_push_size (CLI flag -distributor.max-push-size), default 2 GB. Applies to both the compressed and decompressed size of the request, and is configurable per tenant.
• Per-stream label string size: 16 MB (hardcoded, not configurable)
• Nginx gateway (Helm): gateway.nginxConfig.clientMaxBodySize (default: 4 MB)
• Alloy default batch size: 1 MB

Resolution:

• Split large batches into smaller, more frequent requests.

• Increase the gateway/reverse proxy limit. For example, in the Helm chart set gateway.nginxConfig.clientMaxBodySize; default is 4M.

• Increase limits_config.max_push_size if the error is coming from Loki’s distributor rather than a gateway, and your cluster has the resources to handle larger requests.

• Reduce the number or size of labels on a stream if the error is caused by a single stream’s label string exceeding the hardcoded 16MB limit; this limit cannot be increased.

Properties:

• Enforced by: Gateway/reverse proxy, or the distributor via max_push_size
• Retryable: No
• Sample discarded: Yes
• HTTP status: 413 Request Entity Too Large
• Configurable per tenant: Yes, for max_push_size. No, for gateway/reverse proxy limits or the hardcoded 16MB label-size limit.

Storage errors

Storage errors occur when Loki cannot write data to its storage backend or the Write-Ahead Log (WAL).

Error: Disk space full

Error message:

no space left on device

Cause:

Local storage has run out of space. This can affect ingester WAL, chunk cache, or temporary files.

Default configuration:

Disk space is not limited by Loki configuration; it depends on your infrastructure provisioning.

Resolution:

• Free up disk space on Loki instances.
• Configure retention policies to remove old data.
• Scale storage capacity by adding more disk space.
• Monitor disk usage and set up alerts before reaching critical levels.
• Review compactor settings to ensure old data is being cleaned up.

Properties:

• Enforced by: Operating system
• Retryable: Yes (after freeing space)
• HTTP status: 500 Internal Server Error
• Configurable per tenant: No

Error: Chunk write failure

Error logged:

failed flushing streams (or failed flushing tenant streams when flushing a single tenant via the debug API)

Common causes and errors:

• S3/Object Storage errors:
  • NoSuchBucket: Storage bucket doesn’t exist
  • AccessDenied: Invalid credentials or permissions
  • RequestTimeout: Network or storage latency issues

Resolution:

• Verify storage configuration:

  YAML Copy

  storage_config:
    aws:
      s3: s3://region/bucket-name
      s3forcepathstyle: true

• Check credentials and permissions:

  • Ensure service account has write permissions
  • Verify IAM roles for S3
  • Check API keys for cloud storage

• Monitor storage health:

  • Check object storage availability

• Review storage metrics:

  • loki_ingester_chunks_flushed_total
  • loki_ingester_chunks_flush_failures_total

• Increase retries and timeouts:

  YAML Copy

  storage_config:
    aws:
      s3:
        http_config:
          idle_conn_timeout: 90s
          response_header_timeout: 0s

Properties:

• Enforced by: Ingester (during chunk flush)
• Retryable: Yes (with backoff)
• HTTP status: 500 Internal Server Error
• Configurable per tenant: No

Error: WAL disk full

Error logged:

Error writing to WAL, disk full, no further messages will be logged for this error

Metric: loki_ingester_wal_disk_full_failures_total

Cause:

By default, the ingester monitors disk usage on the volume that stores the WAL and rejects new writes once usage reaches 90% of capacity (-ingester.wal-disk-full-threshold), so that it never accepts data it cannot persist. This error and the associated log message describe a different, rarer case: the disk still filled up completely, either because this throttling is disabled (threshold set to 0), or because a write reached the disk in the short window before the check could catch it. In that case, the ingester continues accepting writes but doesn’t log them to the WAL, losing durability guarantees for those entries.

Default configuration:

• WAL enabled by default
• WAL location: configured by -ingester.wal-dir
• Checkpoint interval: ingester.checkpoint-duration (default: 5 minutes)
• Disk full throttling threshold: ingester.wal-disk-full-threshold (default: 0.9, meaning 90%)

Resolution:

• Increase disk space for the WAL directory.

• Monitor disk usage and set up alerts:

  promql Copy

  loki_ingester_wal_disk_full_failures_total > 0

  This counter increases both when disk usage crosses the throttling threshold and when a WAL write fails because the disk is completely full. It does not count each individual throttled write, so also watch the gauge loki_ingester_wal_disk_usage_percent to see how close the disk is to the threshold.

• Reduce log volume to decrease WAL growth.

• Check WAL checkpoint frequency:

  YAML Copy

  ingester:
    checkpoint_duration: 5m

• Verify WAL cleanup is working - old segments should be deleted after checkpointing.

Properties:

• Enforced by: Ingester
• Retryable: N/A (writes are accepted but not persisted to WAL)
• HTTP status: N/A (writes succeed, but durability is compromised)
• Configurable per tenant: No

Error: WAL corruption

Error logged (at startup, during WAL replay):

• Recovered from checkpoint with errors. Some streams were likely not recovered due to WAL checkpoint file corruptions (or WAL file deletions while Loki is running). No administrator action is needed and data loss is only a possibility if more than (replication factor / 2 + 1) ingesters suffer from this.
• Recovered from WAL segments with errors. Some streams and/or entries were likely not recovered due to WAL segment file corruptions (or WAL file deletions while Loki is running). No administrator action is needed and data loss is only a possibility if more than (replication factor / 2 + 1) ingesters suffer from this.

Metric: loki_ingester_wal_corruptions_total

Cause:

The WAL has become corrupted, possibly due to:

• Disk errors
• Process crashes during write
• Filesystem issues
• Partial deletion of WAL files

Default configuration:

• WAL enabled by default
• Automatic repair attempted on startup
• WAL location: configured by -ingester.wal-dir

Resolution:

• Monitor for corruption:

  promql Copy

  increase(loki_ingester_wal_corruptions_total[5m]) > 0

• Automatic recovery: Loki attempts to recover readable data and continues starting.

• Investigate root cause:

  • Check disk health
  • Review system logs for I/O errors
  • Verify filesystem integrity

Properties:

• Enforced by: Ingester (on startup)
• Retryable: N/A (Loki attempts automatic recovery)
• HTTP status: N/A (occurs during startup, not request handling)
• Configurable per tenant: No

Network and connectivity errors

These errors occur due to network issues or service unavailability.

Error: Connection refused

Error message:

connection refused when connecting to loki:3100

Cause:

The Loki service is unavailable or not listening on the expected port.

Default configuration:

• Loki HTTP listen port: 3100 (-server.http-listen-port)
• Loki gRPC listen port: 9095 (-server.grpc-listen-port)

Resolution:

• Verify Loki is running and healthy.
• Check network connectivity between client and Loki.
• Confirm the correct hostname and port configuration.
• Review firewall and security group settings.
• Check for CPU starvation as a Loki pod that is overwhelmed could fail to respond to a request.

Properties:

• Enforced by: Network/OS
• Retryable: Yes (after service recovery)
• HTTP status: N/A (connection-level error)
• Configurable per tenant: No

Error: Context deadline exceeded

Error message:

context deadline exceeded

Cause:

Requests are timing out due to slow response times or network issues.

Default configuration:

• Alloy default timeout: 10 seconds
• Loki server timeout: -server.http-write-timeout (default: 30s)

Resolution:

• Increase timeout values in your ingestion client.
• Check Loki performance and resource utilization.
• Review network latency between client and server.
• Scale Loki resources if needed.

Properties:

• Enforced by: Client/Server
• Retryable: Yes
• HTTP status: 504 Gateway Timeout (or client-side timeout)
• Configurable per tenant: No

Error: Client canceled request (Code 499)

Error message:

level=warn method=/logproto.Querier/GetChunkIDs duration=<duration> msg=gRPC err="rpc error: code = Code(499) desc = the request was cancelled by the client"

Cause:

A downstream client (such as Grafana, a load balancer, or another Loki component) canceled a gRPC request before the server finished processing it. Code(499) is a non-standard gRPC status code derived from the HTTP 499 status (“Client Closed Request”), which Loki uses internally when it detects a Go context.Canceled error. Because it is not the standard gRPC Canceled code (code 1), the gRPC logging middleware logs it at warn level instead of debug level.

The method field indicates which gRPC operation was interrupted. Common methods include /logproto.Querier/GetChunkIDs, /logproto.Querier/Query, and /logproto.Querier/Series. Although these are query-path operations, the error appears in ingester logs because ingesters serve gRPC read requests from queriers.

Common triggers include:

• A user cancels a running query in Grafana
• A query timeout is reached in the query frontend, query scheduler, or a reverse proxy
• The query frontend cancels redundant split or sharded sub-queries after it has enough results
• A load balancer or ingress controller closes the connection due to its own timeout settings

Default configuration:

• Per-tenant query timeout: query_timeout in limits_config / -querier.query-timeout (default: 1m)
• Loki server HTTP write timeout: -server.http-write-timeout (default: 30s)
• Load balancer / reverse proxy timeout: depends on your infrastructure

Resolution:

• Determine whether this is expected behavior. Occasional Code(499) warnings are normal in production and indicate that a client disconnected before the server responded. These are safe to ignore if they occur infrequently.

• Investigate frequent occurrences. If you see a high rate of Code(499) errors, check for:

  • Queries that are too broad or slow, causing clients to time out
  • Timeout mismatches between layers (for example, a load balancer timeout shorter than the Loki query timeout)
  • Overloaded ingesters or queriers that cannot respond quickly enough

• Align timeouts across your stack. Ensure that timeouts increase from outer layers inward so that Loki has time to respond before upstream components give up:

  Copy

  Grafana timeout > Load balancer timeout > Query frontend timeout > Querier timeout

• Check query performance. Use Loki metrics to identify slow queries:

  promql Copy

  histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, route))

• Reduce log noise. If the warnings are expected and you want to reduce log volume, increase the log level for the affected component to error using -log.level=error. Note that this suppresses all warn-level messages, not just Code(499) warnings.

Properties:

• Enforced by: gRPC logging middleware (dskit)
• Retryable: N/A (the original client is no longer waiting for a response)
• HTTP status: 499 (non-standard, client closed request)
• Configurable per tenant: No

Error: Kafka producer backpressure

These errors occur only when the distributor is configured to write to Kafka (-distributor.kafka-writes-enabled=true), and indicate that the Kafka producer could not keep up with the incoming write volume.

Error messages:

• records have timed out before they were able to be produced
• the maximum amount of records are buffered, cannot buffer more

Cause:

The distributor’s Kafka producer wasn’t able to deliver records to the Kafka brokers within kafka_config.write_timeout, or the amount of unacknowledged, buffered data reached kafka_config.producer_max_buffered_bytes. Both are transient backpressure conditions: the Kafka brokers or the network path to them are slower than the rate at which the distributor is trying to produce records, for example during a broker rollout, a partition rebalance, or a broker-side incident.

Default configuration:

• kafka_config.write_timeout (-kafka.write-timeout): 10s
• kafka_config.producer_max_buffered_bytes (-kafka.producer-max-buffered-bytes): 1 GiB (0 disables the limit)

Resolution:

• Implement retry logic with exponential backoff in your client.

• Check Kafka broker health and load (under-replicated partitions, broker CPU/disk/network saturation, ongoing rollouts).

• Increase kafka_config.write_timeout if brokers are healthy but consistently slow to acknowledge writes:

  YAML Copy

  kafka_config:
    write_timeout: 20s

  Ensure that this remains less than the server’s -server.http-write-timeout (default: 30s), which applies to the distributor’s HTTP server along with every other Loki component, to avoid the server timing out the request first.

Properties:

• Enforced by: Distributor (Kafka producer)
• Retryable: Yes
• HTTP status: 500 Internal Server Error
• Configurable per tenant: No

Error: Service unavailable

Error messages:

• circuit breaker open, request denied — the distributor’s push circuit breaker has tripped because the distributor is overloaded.
• the server cannot accept more requests at this time — the distributor has rejected the request because it is at its configured maximum load.
• Loki can also return 503 when ingesters are unhealthy or the ring is not ready (for example during a rollout or scale-up), without a single fixed error string.

Cause:

Loki is temporarily unable to handle requests due to high load, an open circuit breaker, or maintenance. This can occur when ingesters are unhealthy, the ring is not ready, or the distributor’s push circuit breaker has tripped to protect itself and downstream ingesters from being overwhelmed.

Default configuration:

• Ingester ring readiness: Requires minimum healthy ingesters based on replication factor
• Distributor: Returns 503 when no healthy ingesters are available, when its push circuit breaker is open, or when it is at its configured maximum load

Resolution:

• Implement retry logic with exponential backoff in your client.

• Scale Loki ingester instances to handle load.

• Check for resource constraints (CPU, memory, storage).

• Review ingestion patterns for sudden spikes.

• Check circuit breaker metrics on the distributor to confirm whether the circuit breaker, rather than ring health, is the cause:

  promql Copy

  loki_distributor_circuit_breaker_state
  increase(loki_distributor_circuit_breaker_open_total[5m])

Properties:

• Enforced by: Distributor/Gateway
• Retryable: Yes
• HTTP status: 503 Service Unavailable
• Configurable per tenant: No

Structured metadata errors

These errors occur when using structured metadata incorrectly.

Error: Structured metadata disabled

Error message:

stream '<stream_labels>' includes structured metadata, but this feature is disallowed. Please see limits_config.allow_structured_metadata or contact your Loki administrator to enable it

Reason label: disallowed_structured_metadata

Cause:

allow_structured_metadata defaults to true, so this error usually means it has been explicitly disabled, either globally or as a per-tenant override.

Default configuration:

• allow_structured_metadata: true (enabled by default)

Resolution:

• Enable structured metadata:

  YAML Copy

  limits_config:
    allow_structured_metadata: true

• Move metadata to regular labels if structured metadata isn’t needed.

• Contact your Loki administrator to enable the feature.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: Structured metadata too large

Error message:

stream '{job="app"}' has structured metadata too large: '70000' bytes, limit: '65536' bytes. Please see limits_config.max_structured_metadata_size or contact your Loki administrator to increase it

Reason label: structured_metadata_too_large

Cause:

The structured metadata size exceeds the configured limit.

Default configuration:

• max_structured_metadata_size: 64 KB

Resolution:

• Reduce the size of structured metadata by removing unnecessary fields.

• Increase the limit:

  YAML Copy

  limits_config:
    max_structured_metadata_size: 128KB

• Use compression or abbreviations for metadata values.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Error: Too many structured metadata entries

Error message:

stream '{job="app"}' has too many structured metadata labels: '150', limit: '128'. Please see limits_config.max_structured_metadata_entries_count or contact your Loki administrator to increase it

Reason label: structured_metadata_too_many

Cause:

The number of structured metadata entries exceeds the limit.

Default configuration:

• max_structured_metadata_entries_count: 128

Resolution:

• Reduce the number of structured metadata entries by consolidating fields.

• Increase the limit:

  YAML Copy

  limits_config:
    max_structured_metadata_entries_count: 256

• Combine related metadata into single entries using JSON or other formats.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Blocked ingestion errors

These errors occur when ingestion is administratively blocked.

Ingestion blocked

Error message:

ingestion blocked for user '<tenant>' until '<time>' with status code '260'

Or for policy-specific blocks (reason label blocked_ingestion_policy):

ingestion blocked for user '<tenant>' (policy: '<policy>') until '<time>' with status code '260'

Reason label: blocked_ingestion (or blocked_ingestion_policy for policy-specific blocks)

Cause:

Ingestion has been administratively blocked for the tenant, typically due to policy violations, billing issues, or maintenance. Status code 260 is a custom Loki status code for blocked ingestion.

Default configuration:

• block_ingestion_status_code: 260 (custom status code). If set to 200, Loki silently drops the blocked data without returning an error to the client, although the data is still counted in loki_discarded_samples_total and loki_discarded_bytes_total.
• block_ingestion_until: unset by default. Takes an RFC3339 date; blocks the whole tenant until that date. Normally set per tenant in runtime overrides, since a global value blocks ingestion for every tenant.
• block_ingestion_policy_until: unset by default. A map of policy name to an RFC3339 date, used with policy_stream_mapping to block only streams matching a specific policy.

Resolution:

• Wait until the block expires if a time limit is set.
• Contact your Loki administrator to understand the block reason.
• Review tenant usage patterns and policies.
• Check for policy-specific blocks if using ingestion policies.

Properties:

• Enforced by: Distributor
• Retryable: No (until block is lifted)
• Sample discarded: Yes, unless block_ingestion_status_code is 200, in which case data is silently dropped
• HTTP status: 260 (custom) or configured status code
• Configurable per tenant: Yes

Error: missing_enforced_labels

Error message:

missing required labels '<labels>' for user '<tenant>' for stream '<stream_labels>' (policy: '<policy>')

Cause:

The tenant (or a specific ingestion policy) has been configured to require certain labels on every stream, and the submitted stream is missing one or more of them.

Default configuration:

• enforced_labels: empty list by default (no labels required). Set with validation.enforced-labels to require the same labels on every stream for a tenant.
• policy_enforced_labels: empty map by default. Requires labels only on streams matching a specific policy defined in policy_stream_mapping. The policy * is the global policy, applied to all streams and extended by other policies.

Resolution:

• Add the missing labels to the stream in your log shipping configuration.
• Review enforced_labels/policy_enforced_labels with your Loki administrator if the requirement seems incorrect for your use case.

Properties:

• Enforced by: Distributor
• Retryable: No
• Sample discarded: Yes
• HTTP status: 400 Bad Request
• Configurable per tenant: Yes

Troubleshooting workflow

Follow this workflow when investigating ingestion issues:

• Check metrics:

  promql Copy

  # See which errors are occurring
  sum by (reason) (rate(loki_discarded_samples_total[5m]))
  
  # Identify affected tenants
  sum by (tenant, reason) (rate(loki_discarded_bytes_total[5m]))

• Review logs from distributors and ingesters. Push errors are logged by default with a rate-limited logger (so they don’t flood your logs). This is controlled per tenant, not in limits_config, but in the configs section of the runtime overrides file:

  YAML Copy

  # runtime overrides file
  configs:
    "<tenant>":
      limited_log_push_errors: true # default: true

  Note

  limited_log_push_errors defaults to true, so push errors are already logged in most setups. Set it to false only if you need to see every individual push error without rate limiting (this can be very verbose).

• Test push endpoint:

  Bash Copy

  curl -H "Content-Type: application/json" \
       -XPOST http://loki:3100/loki/api/v1/push \
       --data-raw '{"streams":[{"stream":{"job":"test"},"values":[["'"$(date +%s)000000000"'","test log line"]]}]}'

• Check tenant limits using the overrides-exporter target, which exposes per-tenant limit overrides as Prometheus metrics:

  Bash Copy

  loki -target=overrides-exporter -runtime-config.file=runtime.yaml -config.file=config.yaml -server.http-listen-port=8080
  curl http://localhost:8080/metrics | grep loki_overrides

  For details, refer to Overrides exporter.

  Alternatively, you can check tenant limits using the /config/tenant/v1/limits API. This endpoint requires the tenant_limits_allow_publish setting to be enabled explicitly.

• Review configuration for affected tenant or global limits.

• Monitor system resources:

  • Ingester memory usage
  • Disk space for WAL
  • Storage backend health
  • Network connectivity

Additional resources

• Loki Configuration Reference
• Request Validation and Rate Limits
• Write-Ahead Log Documentation
• Overrides exporter
• Alloy Configuration
• Alloy loki.source.file
• Alloy loki.write
• Multi-tenancy and Limits