Documentation Plugins DynamoDB Troubleshooting

Troubleshoot DynamoDB data source issues

This document provides solutions to common issues you may encounter when configuring or using the DynamoDB data source. For configuration instructions, refer to Configure the DynamoDB data source.

Licensing errors

These errors occur when the DynamoDB plugin license isn’t active or valid.

Plugin health check failed

Symptoms:

• DynamoDB dashboards stop working after a Grafana plan change (for example, upgrading from a free trial to a Pro plan).
• Save & test fails with a health check error.
• The plugin appears installed but queries return errors.

Solutions:

The DynamoDB data source is an Enterprise plugin. It’s included during Enterprise trials but isn’t part of the standard Grafana Cloud Pro plan by default.

1. Verify your Grafana subscription includes the Enterprise plugins add-on. Check your plan details in the Grafana Cloud portal.
2. If the license has expired or is missing, contact Grafana Support or your account representative to add the correct entitlement.
3. After confirming the license, reinstall or re-enable the plugin from Administration > Plugins.
4. Click Save & test on the data source to verify the health check passes.

Authentication errors

These errors occur when credentials are invalid, missing, or don’t have the required permissions.

“missing access key”

Symptoms:

• Save & test fails with a “missing access key” error.
• The data source can’t connect after saving.

Solutions:

1. Verify the Access Key ID field is populated in the data source configuration.
2. If using provisioning, confirm the accessKey field is set in secureJsonData.
3. If you recently upgraded from v1.x to v2.x, re-enter your access key and secret key. Refer to V1 to V2 migration issues for details.

“missing secret key”

Symptoms:

• Save & test fails with a “missing secret key” error.

Solutions:

1. Verify the Secret Access Key field is populated in the data source configuration.
2. If using provisioning, confirm the secretKey field is set in secureJsonData.
3. If you recently upgraded from v1.x to v2.x, re-enter both your access key and secret key.

“missing region”

Symptoms:

• Save & test fails with a “missing region” error.

Solutions:

1. Verify the Default Region field is set in the data source configuration.
2. If using provisioning, confirm defaultRegion is set in jsonData.

“unsupported auth type”

Symptoms:

• Save & test fails with “unsupported auth type” followed by the auth type name (for example, “unsupported auth type default”).
• This commonly occurs when selecting AWS SDK Default, Assume Role, or EC2 IAM Role from the Authentication Provider drop-down.

Possible causes and solutions:

The DynamoDB data source only supports two authentication methods: Access & secret key and Credentials file. All other authentication types shown in the drop-down are provided by the shared AWS connection component but aren’t implemented by this plugin.

Cross-account access

Symptoms:

• Queries fail when trying to access DynamoDB tables in a different AWS account.
• Users expect Assume Role to work for cross-account access.

Solutions:

The DynamoDB data source doesn’t support Assume Role or IRSA (IAM Roles for Service Accounts), which are typically used for cross-account access. As a workaround:

1. Create an IAM user in the target account with dynamodb:PartiQLSelect permissions on the required tables.
2. Generate access keys for that IAM user.
3. Configure the data source using Access & secret key authentication with those credentials.
4. If your organization requires role-based access, consider setting up a proxy service that assumes the role and forwards requests to DynamoDB.

“Access denied” or “Authorization failed”

Symptoms:

• Save & test passes but queries fail with access denied errors.
• Certain tables return errors while others work.

Possible causes and solutions:

Credentials file not found or invalid profile

Symptoms:

• Save & test fails when using Credentials file authentication.
• Error messages mention the credentials file or profile.

Solutions:

1. Verify that ~/.aws/credentials exists on the Grafana server.
2. Confirm the profile name in the data source configuration matches a profile in the credentials file.
3. Check file permissions to ensure the Grafana process can read the credentials file.

Intermittent “Invalid Security Token” errors

Symptoms:

• Queries fail periodically with “Invalid Security Token” or similar authentication errors.
• The errors resolve themselves after refreshing the dashboard or waiting a short time.
• The issue recurs on a regular cycle (for example, every hour).

Solutions:

This typically occurs when using temporary credentials (session tokens) that expire while cached in the plugin’s connection pool. The plugin uses static credentials passed at configuration time and doesn’t automatically refresh expired tokens.

1. If using temporary credentials with a session token, rotate the token before it expires and update the sessionToken field in the data source configuration or provisioning.
2. For long-lived access, use permanent IAM access keys instead of temporary session tokens.
3. If using Credentials file authentication, verify that the credentials file is kept up to date by an external process (for example, aws sso login or an instance profile refresh agent).
4. Restart the Grafana server or data source pods to clear cached connections with stale tokens.
5. Upgrade the DynamoDB plugin to the latest version, as newer releases may include improvements to token handling.

The plugin automatically retries queries that return “Invalid SessionHandle” errors (up to 5 times with a 5-second pause), but this retry mechanism doesn’t cover all token expiration scenarios.

Connection errors

These errors occur when Grafana can’t reach the DynamoDB endpoints.

Connection timeout or refused

Symptoms:

• Save & test times out or fails with a connection error.
• Queries fail intermittently with network errors.

Solutions:

1. Verify network connectivity from the Grafana server to AWS DynamoDB endpoints.
2. Check that firewall rules allow outbound HTTPS (port 443) to dynamodb.<region>.amazonaws.com.
3. If using a custom endpoint, verify the Endpoint field contains the correct URL.
4. For Grafana Cloud accessing private resources, configure Private data source connect.

Custom endpoint not working

Symptoms:

• Queries fail when using a custom endpoint (for example, DynamoDB Local).
• Save & test returns unexpected errors.

Solutions:

1. Verify the endpoint URL is accessible from the Grafana server.
2. Confirm the endpoint URL format is correct (for example, http://localhost:8000 for DynamoDB Local).
3. Check that the region setting is still valid, even when using a custom endpoint.

Configuration errors

These errors occur when the data source configuration is invalid.

“Invalid Settings”

Symptoms:

• Save & test fails with an “Invalid Settings” error followed by a JSON parsing message.
• This typically occurs with provisioned data sources.

Solutions:

1. Verify the jsonData section in your provisioning YAML is valid JSON.
2. Check for syntax errors such as missing quotes, commas, or brackets.
3. Ensure isV2 is set to true in jsonData for new data source configurations.

V1 to V2 migration issues

Symptoms:

• After upgrading the plugin from v1.x to v2.x, a warning appears: “Your plugin has been upgraded from V1 to V2, introducing a breaking change.”
• The data source stops working after editing the configuration page.
• Access key or secret key appears to be missing.

Solutions:

The v2.0.0 release introduced a breaking change to the configuration format. If you edit the configuration page after upgrading, the secret key may be lost.

1. Before editing the configuration, make sure you have both your Access Key ID and Secret Access Key available.
2. Re-enter both credentials on the configuration page.
3. Click Save & test to verify the connection.
4. If using provisioning, update your YAML to use the V2 format with isV2: true, authType, and defaultRegion in jsonData. Refer to Provision the data source for examples.

Query errors

These errors occur when running PartiQL queries against DynamoDB.

“No data” or empty results

Symptoms:

• The query runs without error but returns no data.
• Panels show a “No data” message.

Possible causes and solutions:

Dynamic time filtering doesn’t work

Symptoms:

• Queries with time-based filters return unexpected results or no data.
• Range comparisons such as WHERE "timestamp" > 1711234567 fail or return incorrect results.
• Users expect Grafana variables like $__from or $__timeFilter to work.

Solutions:

The DynamoDB data source doesn’t support time-range macros ($__from, $__to, $__timeFilter, $__range). You must use explicit values or Grafana template variables for time filtering.

Additionally, DynamoDB doesn’t have a native NOW() function, so you can’t dynamically calculate relative time ranges in PartiQL.

A common cause of unexpected results is a mismatch between the timestamp data type in DynamoDB and the comparison value in the query:

To work around the lack of dynamic time filtering:

1. Verify the timestamp column’s data type in DynamoDB matches your query’s comparison format.
2. For alerting, design your DynamoDB data to include only recent entries (for example, using TTL), or accept that alert queries scan a broader range.
3. Consider using a template variable with a custom query to pass time boundaries, though these still require explicit values.

Nested map data returns null

Symptoms:

• Queries that access nested map attributes return null values.
• Top-level attributes in the same query return data correctly.

Solutions:

This is a known limitation of the DynamoDB data source. Querying data from nested maps isn’t supported and returns null values. As a workaround, restructure your data to use top-level attributes, or use DynamoDB Streams to flatten nested data into a separate table.

Query timeout

Symptoms:

• The query runs for a long time and then fails.
• Error messages mention timeout or exceed limits.

Solutions:

1. Narrow the time range to reduce the data volume.
2. Add a partition key filter to your WHERE clause to avoid full table scans.
3. Use LIMIT to restrict the number of returned items.
4. Break complex queries into smaller, more focused queries.
5. Increase the query timeout by setting the timeout field in jsonData through provisioning. The default is 60 seconds. Refer to Advanced settings for details.

Template variable errors

These errors occur when using template variables with the DynamoDB data source.

Variables return no values

Symptoms:

• The variable drop-down is empty.
• The variable preview in Dashboard settings > Variables shows no results.

Solutions:

1. Verify the data source connection is working by running a test in the data source settings.
2. Check that the variable query returns at least one column.
3. Verify the IAM identity has permissions to query the table used in the variable query.
4. For cascading variables (variables that depend on other variables), verify that parent variables have valid selections.

Variables are slow to load

Symptoms:

• Dashboards take a long time to load because of variable queries.
• The variable drop-down is unresponsive while values load.

Solutions:

1. Set variable refresh to On dashboard load instead of On time range change.
2. Add a partition key filter to variable queries to avoid full table scans.
3. Use LIMIT to restrict the number of values returned.

Performance issues

These issues relate to slow queries or API throttling.

DynamoDB throttling or rate limit errors

Symptoms:

• Errors mentioning “503” or throughput exceeded.
• Dashboard panels intermittently fail to load.
• Queries succeed on retry but fail on initial load.

The plugin automatically retries queries that return “503” or “Invalid SessionHandle” errors, up to 5 times with a 5-second pause between attempts. If queries still fail after retries, the underlying throttling needs to be addressed.

Solutions:

1. Reduce the frequency of dashboard auto-refresh.
2. Reduce the number of panels querying DynamoDB simultaneously.
3. Add partition key filters to all queries to avoid full table scans.
4. Increase the provisioned read capacity or switch to on-demand capacity mode in the AWS DynamoDB console.
5. Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud).
6. Adjust retry settings through provisioning if needed. Refer to Advanced settings for the retries and pause configuration options.

Enable debug logging

To capture detailed error information for troubleshooting:

1. Set the Grafana log level to debug in the configuration file:

   ini Copy

   [log]
   level = debug

2. Review logs in /var/log/grafana/grafana.log (or your configured log location).

3. Look for DynamoDB-specific entries that include request and response details.

4. Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions in this document and still encounter issues:

1. Check the Grafana community forums for similar issues.
2. Review the DynamoDB data source GitHub issues for known bugs.
3. Consult the Amazon DynamoDB documentation for service-specific guidance.
4. Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Advanced user.
5. When reporting issues, include:
   • Grafana version and DynamoDB plugin version
   • Error messages (redact sensitive information)
   • Steps to reproduce
   • Relevant configuration (redact credentials)