--- title: "Troubleshoot CloudWatch metric streams | Grafana Cloud documentation" description: "Troubleshoot issues with CloudWatch metric streams for Cloud Provider Observability, including metric stream setup, Amazon Data Firehose delivery failures, and metrics missing from Grafana Cloud." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Troubleshoot CloudWatch metric streams Use this page to troubleshoot issues with CloudWatch metric streams, regardless of whether you configured them using [CloudFormation](../config-cw-metric-streams-cloudformation/) or [Terraform](../config-cw-metric-streams-terraform/). ## CloudWatch metric stream not starting **Problem**: The CloudWatch metric stream shows a failed or inactive state. **Solution**: - Verify the AWS IAM role for the CloudWatch metric stream has permissions to write to Data Firehose. - Check that the Data Firehose delivery stream exists and is active. - If you used CloudFormation, review the CloudFormation stack events for error messages. - If you used Terraform, run `terraform plan` to check for configuration drift, and review the Terraform apply output for errors. ## No metrics appearing in Grafana **Problem**: The CloudWatch metric stream and Data Firehose delivery stream are both active, but no metrics appear in Grafana Cloud. **Solution**: - Verify the Grafana Cloud access policy token used for `metrics_write_token` is correct and has `metric:write` permissions, and that it hasn’t expired. - Check the Data Firehose delivery stream metrics in the AWS console for failed delivery attempts. - Review the Amazon S3 fallback bucket for failed batches. > Tip > > An empty Amazon S3 fallback bucket doesn’t always mean every metric was delivered. Refer to [Metrics are missing but the S3 fallback bucket is empty](#metrics-are-missing-but-the-amazon-s3-fallback-bucket-is-empty). - Ensure the Grafana Cloud AWS Metric Streams Ingest URL (`target_endpoint`) is correct in the Data Firehose configuration. Refer to [Obtain the ingest URL and Username](/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/aws/cloudwatch-metrics/metric-streams/#obtain-the-ingest-url-and-username) for information on how to derive the ingest URL. - If you enabled CloudWatch Logs for delivery errors, review those logs for the underlying failure reason. - It can take 2-3 minutes for the first metrics to appear in Grafana Cloud after the CloudWatch metric stream becomes active. ## Metrics are missing but the Amazon S3 fallback bucket is empty **Problem**: The CloudWatch metric stream and Data Firehose delivery stream both show healthy status, the Amazon S3 fallback bucket is empty, but some metrics are still missing from Grafana Cloud. **Cause**: Amazon Data Firehose retries every delivery response except an HTTP `200`. To avoid retrying deliveries that can never succeed, the Grafana Cloud AWS Metric Streams Ingest URL returns a `200` response to Data Firehose whenever every metric in a batch fails with a non-retryable error from Mimir (a `4xx` response, except `429`). Data Firehose treats that `200` as a successful delivery, so it doesn’t queue the batch to the Amazon S3 fallback bucket, even though Grafana Cloud discarded the data. This most commonly happens when a batch is rejected outright, for example because the batch size exceeds the request limit for Mimir. Amazon Data Firehose also treats a `413` (request size exceeded) response as a permanent failure and doesn’t send that batch to the Amazon S3 fallback bucket, even if backup-on-failure is configured. **Solution**: - If the rejection reason is related to request size, reduce `buffering_size` (Terraform) or `SizeInMBs` (CloudFormation) on the Data Firehose delivery stream. Refer to [Firehose rejects batches with a 413 error](#data-firehose-rejects-batches-with-a-413-error). - Don’t rely solely on an empty Amazon S3 fallback bucket to confirm all metrics were delivered successfully. ## Data Firehose rejects batches with a 413 error **Problem**: Data Firehose logs show `413` errors, or Grafana Cloud logs show `err-mimir-distributor-max-otlp-request-size` rejections. **Cause**: The Grafana Cloud AWS Metric Streams Ingest URL doesn’t split incoming batches before forwarding them to Mimir, so whatever size Data Firehose flushes is sent as a single request. Mimir enforces a 20 MB limit on uncompressed OTLP request size (`-distributor.max-otlp-request-size`). Because OTLP 1.0 payloads are typically 5-7 times larger than the compressed Data Firehose buffer, a buffer above roughly 2 MB risks exceeding this limit. **Solution**: - Keep `buffering_size` (Terraform) or `SizeInMBs` (CloudFormation) at the default of 1 MB. This default is intentional, not a placeholder. - If you need to stream a higher volume of metrics than a 1 MB buffer supports, create additional CloudWatch metric streams split by namespace (or group of namespaces) rather than increasing the buffer size. The Data Firehose HTTP destination doesn’t split oversized batches, so increasing the buffer only increases the risk of hitting the request size limit. ## High Amazon S3 storage costs **Problem**: The Amazon S3 fallback bucket is accumulating data. **Solution**: - Failed batches indicate a delivery problem. Check the Data Firehose error logs. - Verify the Grafana Cloud access policy token hasn’t expired. - Ensure the Grafana Cloud AWS Metric Streams Ingest URL is reachable from your AWS region. - Consider adding Amazon S3 lifecycle policies to delete old failed batches. ## Terraform-specific issues ### Terraform apply fails with IAM errors **Problem**: Terraform can’t create AWS IAM roles or policies. **Solution**: - Verify your AWS credentials have sufficient permissions. Your AWS user or role should have `iam:CreateRole`, `iam:PutRolePolicy`, and related permissions. - Check for AWS IAM role name conflicts if you’re re-creating resources. ### Grafana provider authentication fails **Problem**: Terraform can’t authenticate with the Grafana Cloud API. **Solution**: - Verify the Grafana Cloud access policy token has the required scopes (`integration-management:read`, `integration-management:write`, `stacks:read`). - Check that the Cloud Provider API URL matches your stack’s region. - Ensure the Grafana Cloud access policy token hasn’t expired. ### External ID mismatch error **Problem**: Grafana Cloud can’t assume the resource metadata AWS IAM role. **Solution**: - Ensure `data.grafana_cloud_stack.main.prometheus_user_id` is used as the External ID in the role’s trust policy. - Run `terraform apply` again to update the trust policy. - Verify no manual changes were made to the AWS IAM role in the AWS console.