Troubleshoot Cloud Logs write issues
Unfortunately, occasionally logs are rejected and write failures occur. This can be for a number of reasons; determining the reason for write failures can be challenging.
Because Loki has detailed information about write failures and can emit them through logs, Cloud Logs users can see write errors in a self-service fashion. However, there are limitations to what errors Loki can log. The following are example of errors that cannot be diagnosed with this feature:
- If the push request is too big (for example, 1GB). Grafana cloud would reject the request before it reaches Loki, so it would never be logged.
- Promtail internal issues, for example, Promtail running out of memory. Promtail issues wouldn’t be discovered because the request would never reach the Grafana Cloud to be logged by Loki.
To troubleshoot logs write issues:
- Log in to your GrafanaCloud instance.
- Click the menu icon, then click the Explore (compass) icon.
- From the menu in the top left of the Explore page, select the
grafanacloud-[instanceName]-usage-insights
datasource. - Query the datasource for write issues.
- Use the Query Builder on the Builder tab.
- Or, click the Code tab and enter your query in the field. Example queries:
{instance_type="logs"} |= "path=write"
{instance_type="logs"} |= "push request failed"
The query returns log messages, which include information to help determine why the log write failed. The log messages include which stream discarded logs, for what reason, and provides valid values.
Note
You can also troubleshoot write issues with Dashboards.
The following sections contain the errors that can be discovered using this feature and the message displayed in the logs.
Entry too far behind
Entry too far behind: “entry too far behind, oldest acceptable timestamp is: %s”
This validation error is returned when a stream is submitted out of order. This problem can be solved by ensuring that log delivery is configured correctly. It should also be noted that this a per-stream error, so by simply splitting streams (adding more labels) this problem can be circumvented, especially if multiple hosts are sending samples for a single stream.
Entry too far in the future
Entry too far in the future: “entry for stream ‘%s’ has timestamp too new: %v”
This error can be solved by increasing the creation_grace_period
value, or investigating why this particular stream has a timestamp too far into the future. The stream in question will be returned in the body of the HTTP response.
Entry too old
Entry too old: “entry for stream ‘%s’ has timestamp too old: %v, oldest acceptable timestamp is: %v”
This error means that the timestamp for the log is greater than the reject_old_samples_max_age
value; the default value is one week.
Duplicate label names
Duplicate label names: “stream ‘%s’ has duplicate label name: ‘%s’”
This error can be solved by removing one of the identical labels.
Invalid labels
Invalid labels: “Error parsing labels ‘%s’ with error: %s”
Check that your labels are valid. Label names may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*
. Label names beginning and ending with double underscores __
are reserved for internal use.
Label name too long
Label name too long: “stream ‘%s’ has label name too long: ‘%s’”
The default value for max_label_name_length
is 1024 characters. You can use logcli to print a list of all your labels and their unique values: logcli series '{cluster="vinson"}' --analyze-labels
.
Label value too long
Label value too long: “stream ‘%s’ has label value too long: ‘%s’”
The default value for max_label_value_length
is 2048 characters. This setting also applies to the metric name. Check the lengths of your label names, or remove one or more labels from the log stream.
Line too long
Line too long: “Max entry size ‘%d’ bytes exceeded for stream ‘%s’ while adding an entry with length ‘%d’ bytes”
The default value for max_line_size
is 256k
. Either reconfigure your client to truncate log lines that are too long, or contact Support to set distributor.max-line-size-truncate
to true
.
Max label names per series
Max label names per series: “entry for stream ‘%s’ has %d label names; limit %d”
The maximum label names per series is 15. This error can be solved by removing one or more labels until you are below the limit.
Missing labels
Missing labels: “error at least one label pair is required per stream”
Add at least one label to your logs to define a log stream.
Rate limited
Rate limited: “Ingestion rate limit exceeded for user %s (limit: %d bytes/sec) while attempting to ingest ‘%d’ lines totaling ‘%d’ bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased”
Contact Grafana Support to see if the limit can be increased.
Stream limit
Stream limit: “Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased”
Contact Grafana Support to see if the limit can be increased.
Note
For Grafana Cloud Logs customers, the “Loki administrator” is Grafana Support. If you would like to inquire about increasing a rate limit, please file a support ticket.