Documentation Plugins OpenSearch Troubleshooting

Troubleshoot OpenSearch data source issues

This document provides solutions to common issues you might encounter when configuring or using the OpenSearch data source. For configuration instructions, refer to Configure the OpenSearch data source.

Authentication errors

These errors occur when credentials are invalid, missing, or don’t have the required permissions.

“Access denied” or authorization errors

Symptoms:

• Save & test fails with authorization errors
• Queries return access denied messages
• Fields or indices don’t load in drop-downs

Possible causes and solutions:

SigV4 authentication failures

Symptoms:

• Save & test fails when using AWS SigV4 authentication
• Error messages related to signature validation or authorization

Possible causes and solutions:

Connection errors

These errors occur when Grafana can’t reach the OpenSearch instance.

“Connection refused” or timeout errors

Symptoms:

• Data source test times out
• Queries fail with network errors
• Intermittent connection issues

Solutions:

1. Verify the URL in the data source settings is correct, including the protocol and port.
2. Check that the Grafana server can reach the OpenSearch instance over the network.
3. Verify firewall rules allow outbound traffic on the configured port.
4. For private networks, ensure VPN or private connectivity is configured.
5. For Grafana Cloud, configure Private data source connect if accessing private resources.

TLS or certificate errors

Symptoms:

• Errors related to certificate verification or TLS handshake failures
• “x509: certificate signed by unknown authority” errors

Solutions:

1. If using a self-signed certificate, provide the CA certificate in the TLS/SSL Auth Details section.
2. For testing, enable Skip TLS Verify in the data source settings (not recommended for production).
3. Ensure the certificate isn’t expired.
4. Verify the certificate’s Subject Alternative Name (SAN) matches the hostname in the URL.

Configuration errors

These errors occur during data source setup or health checks.

“No version set”

Symptoms:

• Save & test fails with “No version set”
• The version field is empty in the data source settings

Solutions:

1. Click Save & test to let the plugin auto-detect the OpenSearch version.
2. If auto-detection fails, verify the URL is correct and the OpenSearch instance is reachable.
3. Ensure the credentials have permissions to query the root endpoint.

“ElasticSearch version is not supported by the OpenSearch plugin. Use the ElasticSearch plugin.”

Symptoms:

• Save & test returns “ElasticSearch version X.Y.Z is not supported by the OpenSearch plugin. Use the ElasticSearch plugin.”

Solutions:

This error occurs when the plugin detects an Elasticsearch instance with version 7.11 or later. Install and use the Elasticsearch data source instead of the OpenSearch data source.

“Generated empty index list”

Symptoms:

• Save & test fails with “Generated empty index list”

Solutions:

This error occurs when using a time-based index pattern and no indices match the pattern within the health check window (the last six hours).

1. Verify the index name pattern and Pattern interval are correct.
2. Ensure indices exist that match the pattern for the most recent six-hour window.
3. If indices only exist for older data, use a static index name instead of a time-based pattern.

“Unable to fetch fields from the datasource”

Symptoms:

• Drop-down menus for fields don’t populate
• Error includes details from an OpenSearch root_cause response

Solutions:

1. Verify the index name is correct and the index exists.
2. Ensure the credentials have permissions to call the _field_caps API on the target index.
3. Check that the OpenSearch instance is reachable.

Query errors

These errors occur when executing queries against OpenSearch.

“No data” or empty results

Symptoms:

• Query executes without error but returns no data
• Panels show a “No data” message

Possible causes and solutions:

“Index not found”

Symptoms:

• Save & test returns “Index not found” errors
• Queries fail with index-related errors

Solutions:

1. Verify the index name in the data source settings exists in OpenSearch.
2. If using an index pattern with a time-based suffix, verify the Pattern setting matches your naming convention.
3. Check that the index hasn’t been deleted or rolled over.
4. For time-based patterns, ensure data exists within the dashboard’s time range.

“invalid queryType, expected Lucene or PPL”

Symptoms:

• Query fails with invalid queryType: "<value>", expected Lucene or PPL

Solutions:

1. Verify the query type is set to either Lucene or PPL in the query editor.
2. If using provisioned dashboards, ensure the queryType field in the query JSON is set to "lucene" or "ppl".

PPL query errors

Symptoms:

• PPL queries fail with syntax errors or an HTTP status code error from the OpenSearch PPL engine
• Error includes the status code and an OpenSearch reason message

Solutions:

1. Verify PPL is enabled in the data source settings.
2. Check the PPL syntax against the OpenSearch PPL documentation.
3. Verify the index name in the source command matches an existing index.
4. For OpenSearch Serverless, ensure PPL is supported for your collection type.

PPL time-series parsing errors

Symptoms:

• PPL query with Time series format fails with one of the following errors:
  • “response should have 2 fields but found N”
  • “found non-numerical value in value field”
  • “a valid time field type was not found in response”
  • “unable to parse time field”

Solutions:

1. Ensure the PPL query returns exactly two columns: a timestamp field and a numeric value field. Use stats to aggregate and limit to two columns.
2. Verify the timestamp column has a type of timestamp, datetime, or date in the OpenSearch index mapping.
3. Verify the value column contains only numeric data.
4. Test the query in OpenSearch Dashboards or the PPL CLI to inspect the returned schema.

Trace and service map errors

Symptoms:

• Trace queries return errors related to service map fetching
• Error messages such as “Error fetching service map info”
• Span start times or event times fail to parse

Solutions:

1. Ensure the Service Map toggle is enabled only when your data includes the required service map fields.
2. Verify trace data in the target index follows the expected Data Prepper schema.
3. Check that startTime and event time fields are in a parsable format.

Query timeout

Symptoms:

• Query runs for a long time then fails
• Error mentions timeout or query limits

Solutions:

1. Narrow the dashboard time range to reduce data volume.
2. Add filters to the query to reduce the result set.
3. Increase the Max concurrent shard requests setting if queries involve many shards.
4. Break complex aggregation queries into smaller parts.

Template variable errors

These errors occur when using template variables with the data source.

Variables return no values

Solutions:

1. Verify the data source connection is working using Save & test.
2. Check that the JSON query syntax is valid, for example {"find": "terms", "field": "@hostname"}.
3. Ensure the field name in the query exists in the index and has the correct type.
4. For cascading variables, verify that parent variables have valid selections.
5. Check that the credentials have permissions to query the target index.

“Invalid query” or sort errors

Symptoms:

• Template variable query fails with “Invalid query”
• Error mentions an invalid query sort order or sort type

Solutions:

1. Ensure the JSON includes valid find and field properties.
2. If using orderBy, set it to "term", "key", or "doc_count".
3. If using order, set it to "asc" or "desc".

Variables are slow to load

Solutions:

1. Set the variable refresh to On dashboard load instead of On time range change if the variable values don’t change with time.
2. Add a size property to limit the number of returned values, for example {"find": "terms", "field": "@hostname", "size": 100}.
3. Narrow the dashboard time range to reduce the data scanned by the terms aggregation.

Annotation errors

These errors occur when querying OpenSearch for annotation data.

“Error querying annotations”

Symptoms:

• Annotations don’t appear on dashboard graphs
• Error notification shows “Error querying annotations” followed by a reason

Solutions:

1. Verify the data source connection is working using Save & test.
2. Check that the Lucene query in the annotation configuration is syntactically valid.
3. Ensure the Time field is mapped to a valid date field in your index (default: @timestamp).
4. Verify the credentials have read access to the target index.

Annotations don’t support PPL

If you’ve configured an annotation query using PPL syntax, the query fails because annotations only support Lucene queries. Rewrite the annotation query using Lucene syntax.

Alerting errors

These errors occur when creating or evaluating alert rules with OpenSearch queries.

Unsupported query type for alerting

Symptoms:

• Alert rule evaluation returns no data or an error
• Alert preview shows unexpected results

Solutions:

Only Lucene Metric and PPL Time series query types reliably produce numeric time-series data for alerting. Logs, Raw Data, Raw Document, and Traces query types return data in formats that can’t be evaluated by alert conditions. Refer to Supported query types for details.

PPL alert query fails with format errors

Symptoms:

• Alert evaluation fails with errors such as “response should have 2 fields” or “found non-numerical value in value field”

Solutions:

1. Change the Format drop-down from Table (the default) to Time series in the query editor.

2. Ensure the PPL query returns exactly two columns: a timestamp field and a numeric value field.

3. Use stats with a time aggregation to structure the output correctly, for example:

   Copy

   source = my_index | eval dateValue = timestamp(timestamp) | stats count(response) by dateValue

Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to debug in the configuration file:

   ini Copy

   [log]
   level = debug

2. Review logs in /var/log/grafana/grafana.log or your configured log location.

3. Look for OpenSearch-specific entries that include request and response details.

4. Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions in this document and still encounter issues:

1. Check the Grafana community forums for similar issues.
2. Review the OpenSearch plugin GitHub issues for known bugs.
3. Consult the OpenSearch documentation for service-specific guidance.
4. Contact Grafana Support if you have a Grafana Cloud Pro or Advanced plan, or a Grafana Enterprise license.
5. When reporting issues, include:
   • Grafana version and plugin version
   • Error messages (redact sensitive information)
   • Steps to reproduce
   • Relevant configuration (redact credentials)