Documentation Plugins Amazon Timestream Troubleshooting

Troubleshoot Amazon Timestream data source issues

This document provides solutions to common issues you may encounter when configuring or using the Amazon Timestream data source. For configuration instructions, refer to Configure the Amazon Timestream data source.

Minimum Grafana version

The Amazon Timestream data source requires Grafana 10.4 or later. If you encounter unexpected errors, verify that your Grafana version meets this requirement.

Authentication errors

These errors occur when credentials are invalid, missing, or don’t have the required permissions.

“AccessDeniedException” or “Authorization failed”

Symptoms:

• Save & test fails with AccessDeniedException.
• Queries return access denied messages.
• Database, table, or measure drop-downs don’t load.

Possible causes and solutions:

“ExpiredTokenException” or “Token has expired”

Symptoms:

• Queries fail intermittently with token expiration errors.
• The data source works initially but stops after a period of time.

Solutions:

1. If you use temporary credentials, verify that the session token is current.
2. For assume role configurations, verify that the source identity has permission to call sts:AssumeRole.
3. Check that the Grafana server clock is synchronized. Expired token errors can occur when the system clock drifts.

Connection errors

These errors occur when Grafana can’t reach the Timestream API endpoints.

“Connection refused” or timeout errors

Symptoms:

• Save & test times out.
• Queries fail with network errors.
• Intermittent connection failures.

Solutions:

1. Verify network connectivity from the Grafana server to Timestream endpoints in the configured region.
2. Check that firewall rules allow outbound HTTPS traffic on port 443.
3. If you use a VPC endpoint, verify the endpoint is correctly configured and set the custom endpoint URL in the data source settings.
4. For Grafana Cloud, configure Private data source connect if accessing private resources.

Custom endpoint issues

Symptoms:

• Queries fail after configuring a custom endpoint.
• 404 errors related to DescribeEndpoints.
• Errors related to endpoint discovery.

Solutions:

1. Verify the custom endpoint URL is correct and reachable from the Grafana server.
2. When a custom endpoint is configured (for example, a VPC endpoint), the plugin automatically disables AWS endpoint discovery. This prevents 404 errors from VPC endpoints that don’t implement the DescribeEndpoints API. Verify that the endpoint URL points directly to the Timestream query endpoint.
3. If the VPC endpoint blocks egress to public AWS endpoints, ensure endpoint discovery is not being forced by another configuration.
4. To revert to the default endpoint, clear the Default Endpoint field in the data source configuration.

Query errors

These errors occur when executing queries against Timestream.

“No data” or empty results

Symptoms:

• Queries run without error but return no data.
• Panels display a “No data” message.

Possible causes and solutions:

Query timeout

Symptoms:

• Queries run for a long time and then fail.
• Errors mention timeout or query limits.

Solutions:

1. Narrow the dashboard time range to reduce the volume of data scanned.
2. Add filters to the WHERE clause to reduce the result set.
3. Use bin(time, <interval>) with a larger interval to reduce the number of returned rows.
4. Break complex queries into smaller parts using multiple panels.

Macros not interpolating correctly

Symptoms:

• Query contains literal $__timeFilter or $__timeFrom text instead of expanded values.
• Time range filters don’t work after a Grafana upgrade.
• ${__from:date:iso} or other formatted time variables aren’t replaced.

Possible causes and solutions:

Alerting errors

These errors occur when using the Amazon Timestream data source with Grafana Alerting. For setup instructions, refer to Alerting.

“input data must be a wide series but got type long”

Symptoms:

• Alert rules fail with this error.
• The query works in dashboard panels but fails in alerting.

Solutions:

1. Use the CREATE_TIME_SERIES function to return data in wide time-series format. Refer to Alerting for examples.
2. Enable Wait for all queries in the query editor to ensure all result pages are processed before the alert evaluates.

Alert fails on string or non-numeric data

Symptoms:

• Alert rule returns an error about unsupported data types.
• Query returns string values like "HEALTHY" or "RUNNING" that can’t be used as alert conditions.

Solutions:

1. Use a CASE expression to convert string values to numeric values inside CREATE_TIME_SERIES. For example, map 'UNHEALTHY' to 1.0 and 'HEALTHY' to 0.0, then set a threshold on the numeric result.
2. Refer to Alert on string values for a full example.

Pagination and incomplete results

Symptoms:

• Only a subset of expected results appears.
• Results differ between dashboard panels and alerting.

Solutions:

1. Enable Wait for all queries in the query editor to fetch all result pages before returning data.
2. If you don’t enable this option, the plugin streams pages incrementally. This works for dashboard panels but can cause incomplete results in alerting evaluations.

Template variable errors

These errors occur when using template variables with the data source.

Variables return no values

Symptoms:

• Variable drop-downs are empty.
• The variable preview shows no results when you click Run query.

Solutions:

1. Verify the data source connection is working by running Save & test in the data source settings.
2. Check that the variable query is valid SQL that returns at least one column.
3. For cascading variables (where one depends on another), verify that the parent variable has a valid selection.
4. Verify the IAM identity has permission to list databases, tables, or run the specific query used by the variable.

Variables are slow to load

Solutions:

1. Set the variable refresh to On dashboard load instead of On time range change to avoid reloading on every time range adjustment.
2. Simplify variable queries. For example, use SHOW DATABASES instead of a full SELECT DISTINCT query when listing databases.
3. Reduce the scope of variable queries by adding filters.

Performance and cost issues

These issues relate to slow queries, high AWS costs, or API limits. For detailed optimization guidance, refer to Optimize query performance and cost.

High query costs

Symptoms:

• Unexpectedly high AWS Timestream charges.
• Queries scan more data than expected.

Solutions:

1. Always include a measure_name filter in queries. Timestream partitions data by measure name, so omitting this filter scans the entire table.
2. Narrow the dashboard time range to reduce the volume of data scanned.
3. Use bin() with $__interval_ms to aggregate data instead of returning raw rows.
4. Reduce the dashboard auto-refresh frequency.
5. Use the Dashboard data source to reuse query results across multiple panels instead of running duplicate queries.
6. Enable query caching in Grafana Enterprise or Grafana Cloud.

API throttling or rate limit errors

Symptoms:

• “Rate exceeded” or throttling errors.
• Dashboard panels intermittently fail to load.
• Multiple panels fail simultaneously.

Solutions:

1. Reduce the frequency of dashboard auto-refresh.
2. Use larger time intervals in bin() to reduce the number of data points per query.
3. Use the Dashboard data source to share query results across panels instead of running separate queries.
4. Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud).
5. Request a quota increase from AWS through the Service Quotas console.

Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to debug in the configuration file:

   ini Copy

   [log]
   level = debug

2. Review logs in /var/log/grafana/grafana.log or your configured log location.

3. Look for entries containing timestream for request and response details.

4. Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried these solutions and still encounter issues:

1. Check the Grafana community forums for similar issues.
2. Review the Timestream plugin GitHub issues for known bugs.
3. Refer to the Amazon Timestream documentation for service-specific guidance.
4. Contact Grafana Support if you’re a Cloud Pro, Cloud Advanced, or Enterprise user.
5. When reporting issues, include:
   • Grafana version and plugin version
   • Error messages (redact sensitive information)
   • Steps to reproduce
   • Relevant configuration (redact credentials)