Documentation Grafana documentation Data sources Loki Troubleshooting

Troubleshoot Loki data source issues

This document provides solutions to common issues you might encounter when configuring or using the Loki data source. For configuration instructions, refer to Configure the Loki data source.

Connection errors

These errors occur when Grafana can’t reach your Loki server.

Save & test fails

Symptoms:

• The data source test fails or times out.
• Queries fail with network errors.

Possible causes and solutions:

Query errors

These errors occur when running queries against Loki.

No data or empty results

Symptoms:

• A query runs without error but returns no data.
• Panels show a “No data” message.

Possible causes and solutions:

Query timeout or “too many outstanding requests”

Symptoms:

• A query runs for a long time and then fails.
• Errors mention timeouts, query limits, or too many outstanding requests.

Solutions:

1. Narrow the time range to reduce the volume of data scanned.
2. Add label filters or a line filter to reduce the result set.
3. Lower the Maximum lines setting on the data source or the line limit on the query.
4. Review the query and rate limits configured on your Loki deployment.

“maximum of series reached for a single query”

Symptoms:

• A metric query fails with an error about the maximum number of series.

Solutions:

1. Add label matchers to reduce the number of streams the query touches.
2. Aggregate with sum by (...) to reduce cardinality.
3. Avoid high-cardinality labels in aggregations.

Performance issues

These issues relate to slow queries or rate limits.

Queries are slow or get rate limited

Solutions:

1. Narrow the time range and add label filters so queries scan less data.
2. Enable query caching, available in Grafana Enterprise and Grafana Cloud.
3. Review the query, ingestion, and rate limits on your Loki deployment.

Template variable errors

These errors occur when using template variables with the data source.

Variables return no values

Solutions:

1. Confirm the data source connection works by running Save & test.
2. Verify the label or stream selector in the variable query returns values in the query editor.
3. For cascading variables, confirm parent variables have valid selections.

Derived field and log-to-trace link issues

These issues affect derived fields, including the links from logs to traces.

The derived field or trace link doesn’t appear

• Confirm the regular expression matches your log line format. A pattern that doesn’t match produces no value and no link. Use Show example log message in the data source configuration to test the pattern against a real log line.
• For a Label type derived field, confirm a label key matches the pattern. The match runs against label keys, not values.
• Check that the field value is captured. For a Regex in log line type, Grafana uses the first capture group, so the expression must include one.

The trace link opens but shows no trace

• The derived field only builds a link to the target data source using the extracted value; it doesn’t create the trace. Confirm the trace is ingested into the target data source, such as Tempo, and that the extracted ID matches the trace ID stored there.
• Verify the extracted value is the correct ID. An overly permissive regular expression can capture the wrong part of the log line. Anchor the pattern and make it specific, for example match traceID=(\w+) rather than a broad (\w+).

Can’t edit derived fields on a provisioned data source

Data sources created through provisioning are read-only in the UI, so you can’t change their derived fields from the data source configuration page.

• For self-managed Grafana, edit the derived field configuration in the provisioning YAML file and restart Grafana, or reload provisioning.
• For Grafana Cloud, open a support ticket from the Grafana Cloud Portal to request changes to a provisioned data source.

A single link needs values from more than one field

A derived field produces a single value per log line, so you can’t combine multiple labels or capture groups into one field or link. Create a separate derived field for each value you need to surface.

Live tailing issues

These issues affect real-time log tailing in Explore.

Live tailing doesn’t stream logs

Live tailing relies on WebSocket connections between the browser and Grafana, and between Grafana and Loki. Reverse proxies often block these connections.

Solutions:

1. Confirm your reverse proxy forwards WebSocket connections for the tail endpoint. Refer to the proxy examples in the Loki query editor.
2. Verify that intermediate load balancers and firewall rules allow WebSocket upgrades.
3. Confirm Loki itself is reachable and that the tail endpoint isn’t blocked.

Label-based access control (LBAC) issues

Label-based access control (LBAC) filters the logs a team can query based on labels. It’s available on Grafana Cloud and Grafana Enterprise, where it requires Grafana Enterprise Logs (GEL). If you can’t enable LBAC, or teams see more logs than expected, check the following.

Can’t enable LBAC on a provisioned data source

To use LBAC, create a new Loki data source manually and configure it with basic authentication, using your Cloud Access Policy (CAP) token as the password. Point it at the same Loki endpoint as the provisioned data source, then define LBAC rules on the new data source’s Permissions tab. For the full procedure, refer to Configure LBAC for data sources for Loki.

Teams can still query all logs

• LBAC applies only when a team has at least one rule. If a team has no LBAC rules, its members can query all logs. Confirm each team has the rules you intend.
• Grafana permissions are additive. A user’s basic role, such as Viewer, can grant access to the data source independently of team-scoped rules. Remove role and team permissions that aren’t required so that only the intended teams have Query permission on the LBAC data source.
• Remove any label selectors from the Cloud Access Policy used by the data source. CAP label selectors override LBAC rules.

For details and limitations, refer to LBAC for data sources.

Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to debug in the configuration file:

   ini Copy

   [log]
   level = debug

2. Review logs in /var/log/grafana/grafana.log, or in your configured log location.

3. Look for Loki-specific entries that include request and response details.

4. Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried these solutions and still encounter issues:

1. Check the Grafana community forums for similar issues.
2. Review the Loki data source GitHub issues for known bugs.
3. Consult the Loki documentation for service-specific guidance.
4. When reporting issues, include:
   • Grafana and Loki versions
   • Error messages, with sensitive information redacted
   • Steps to reproduce
   • Relevant configuration, with credentials redacted