Documentation Grafana Cloud Platform Security and account management Security and access Manage secrets Troubleshoot AWS keeper

Troubleshoot an AWS Secrets Manager keeper

Use the following sections to resolve common issues and learn about operational considerations for the AWS Secrets Manager keeper. For setup instructions, refer to Configure an AWS Secrets Manager keeper.

Cannot create secrets

Cause: The permissions policy is missing or incomplete.

Solution:

1. Verify GrafanaSecretsManagerPolicy is attached to your IAM role.
2. Check that the policy includes secretsmanager:CreateSecret.
3. Verify the policy resource allows Grafana to create secrets under the grafana-secrets-manager/* path.

Cannot read existing secrets

Cause: The ReadExistingSecrets statement doesn’t allow access to the secret.

Solution:

1. Check if the secret Amazon Resource Name (ARN) matches the Resource in your policy.
2. If you use tag-based conditions, verify the secret has the required tags.
3. Test access using the AWS CLI.

For more information about restricting access, refer to Restrict access to specific secrets.

Operational considerations

Understand the secret retrieval mechanism

Grafana doesn’t cache secret values. Instead, it makes a live GetSecretValue API call to AWS Secrets Manager every time a secret is accessed. Any changes to a secret in AWS, including rotations, are immediately reflected the next time Grafana reads the secret.

Use multiple keepers

You can create multiple keepers with different IAM roles or regions. For example, you might use separate keepers to isolate production and development secrets, access secrets across multiple AWS regions, or assign different IAM roles to different teams.

Remove a keeper from Grafana

Removing a keeper deletes the keeper configuration from Grafana. It doesn’t delete any secrets from AWS Secrets Manager. Value-based secrets that Grafana created remain in your AWS account. To clean them up, manually delete them from AWS Secrets Manager using the AWS Console or CLI. You can identify Grafana-managed secrets by the managedBy tag.

Reference-based secrets are unaffected in AWS because Grafana never owns or modifies those secrets.

Handle AWS Secrets Manager outages

Grafana makes a live API call to AWS Secrets Manager every time a secret is accessed. If AWS Secrets Manager is unavailable due to a regional outage, Grafana can’t retrieve secret values until the service is restored. Any Grafana features that depend on those secrets are affected during the outage.

To reduce risk, consider storing secrets in multiple AWS regions using separate keepers.