Documentation Grafana Cloud Observe and act Monitor infrastructure Cloud Provider Observability AWS observability CloudWatch metrics Configure CloudWatch metric streams Troubleshoot metric streams

Troubleshoot CloudWatch metric streams

Use this page to troubleshoot issues with CloudWatch metric streams, regardless of whether you configured them using CloudFormation or Terraform.

CloudWatch metric stream not starting

Problem: The CloudWatch metric stream shows a failed or inactive state.

Solution:

• Verify the AWS IAM role for the CloudWatch metric stream has permissions to write to Data Firehose.
• Check that the Data Firehose delivery stream exists and is active.
• If you used CloudFormation, review the CloudFormation stack events for error messages.
• If you used Terraform, run terraform plan to check for configuration drift, and review the Terraform apply output for errors.

No metrics appearing in Grafana

Problem: The CloudWatch metric stream and Data Firehose delivery stream are both active, but no metrics appear in Grafana Cloud.

Solution:

• Verify the Grafana Cloud access policy token used for metrics_write_token is correct and has metric:write permissions, and that it hasn’t expired.

• Check the Data Firehose delivery stream metrics in the AWS console for failed delivery attempts.

• Review the Amazon S3 fallback bucket for failed batches.

  Tip

  An empty Amazon S3 fallback bucket doesn’t always mean every metric was delivered. Refer to Metrics are missing but the S3 fallback bucket is empty.

• Ensure the Grafana Cloud AWS Metric Streams Ingest URL (target_endpoint) is correct in the Data Firehose configuration. Refer to Obtain the ingest URL and Username for information on how to derive the ingest URL.

• If you enabled CloudWatch Logs for delivery errors, review those logs for the underlying failure reason.

• It can take 2-3 minutes for the first metrics to appear in Grafana Cloud after the CloudWatch metric stream becomes active.

Metrics are missing but the Amazon S3 fallback bucket is empty

Problem: The CloudWatch metric stream and Data Firehose delivery stream both show healthy status, the Amazon S3 fallback bucket is empty, but some metrics are still missing from Grafana Cloud.

Cause: Amazon Data Firehose retries every delivery response except an HTTP 200. To avoid retrying deliveries that can never succeed, the Grafana Cloud AWS Metric Streams Ingest URL returns a 200 response to Data Firehose whenever every metric in a batch fails with a non-retryable error from Mimir (a 4xx response, except 429). Data Firehose treats that 200 as a successful delivery, so it doesn’t queue the batch to the Amazon S3 fallback bucket, even though Grafana Cloud discarded the data.

This most commonly happens when a batch is rejected outright, for example because the batch size exceeds the request limit for Mimir. Amazon Data Firehose also treats a 413 (request size exceeded) response as a permanent failure and doesn’t send that batch to the Amazon S3 fallback bucket, even if backup-on-failure is configured.

Solution:

• If the rejection reason is related to request size, reduce buffering_size (Terraform) or SizeInMBs (CloudFormation) on the Data Firehose delivery stream. Refer to Firehose rejects batches with a 413 error.
• Don’t rely solely on an empty Amazon S3 fallback bucket to confirm all metrics were delivered successfully.

Data Firehose rejects batches with a 413 error

Problem: Data Firehose logs show 413 errors, or Grafana Cloud logs show err-mimir-distributor-max-otlp-request-size rejections.

Cause: The Grafana Cloud AWS Metric Streams Ingest URL doesn’t split incoming batches before forwarding them to Mimir, so whatever size Data Firehose flushes is sent as a single request. Mimir enforces a 20 MB limit on uncompressed OTLP request size (-distributor.max-otlp-request-size). Because OTLP 1.0 payloads are typically 5-7 times larger than the compressed Data Firehose buffer, a buffer above roughly 2 MB risks exceeding this limit.

Solution:

• Keep buffering_size (Terraform) or SizeInMBs (CloudFormation) at the default of 1 MB. This default is intentional, not a placeholder.
• If you need to stream a higher volume of metrics than a 1 MB buffer supports, create additional CloudWatch metric streams split by namespace (or group of namespaces) rather than increasing the buffer size. The Data Firehose HTTP destination doesn’t split oversized batches, so increasing the buffer only increases the risk of hitting the request size limit.

High Amazon S3 storage costs

Problem: The Amazon S3 fallback bucket is accumulating data.

Solution:

• Failed batches indicate a delivery problem. Check the Data Firehose error logs.
• Verify the Grafana Cloud access policy token hasn’t expired.
• Ensure the Grafana Cloud AWS Metric Streams Ingest URL is reachable from your AWS region.
• Consider adding Amazon S3 lifecycle policies to delete old failed batches.

Terraform-specific issues

Terraform apply fails with IAM errors

Problem: Terraform can’t create AWS IAM roles or policies.

Solution:

• Verify your AWS credentials have sufficient permissions. Your AWS user or role should have iam:CreateRole, iam:PutRolePolicy, and related permissions.
• Check for AWS IAM role name conflicts if you’re re-creating resources.

Grafana provider authentication fails

Problem: Terraform can’t authenticate with the Grafana Cloud API.

Solution:

• Verify the Grafana Cloud access policy token has the required scopes (integration-management:read, integration-management:write, stacks:read).
• Check that the Cloud Provider API URL matches your stack’s region.
• Ensure the Grafana Cloud access policy token hasn’t expired.

External ID mismatch error

Problem: Grafana Cloud can’t assume the resource metadata AWS IAM role.

Solution:

• Ensure data.grafana_cloud_stack.main.prometheus_user_id is used as the External ID in the role’s trust policy.
• Run terraform apply again to update the trust policy.
• Verify no manual changes were made to the AWS IAM role in the AWS console.