Documentation Grafana Cloud Monitor infrastructure Cloud Provider Observability AWS observability CloudWatch metrics Configure CloudWatch metric streams Configure CloudWatch metric streams with CloudFormation

Configure CloudWatch metric streams with CloudFormation

This guide walks you through configuring CloudWatch metric streams using AWS CloudFormation, which includes the following tasks:

1. Launch the CloudFormation stack in AWS
2. Configure optional settings
3. Create the AWS resource metadata scrape job in Grafana Cloud

Launch the CloudFormation stack

The CloudFormation template creates all required AWS resources including IAM roles, the Data Firehose delivery stream, and the CloudWatch metric stream.

To launch the CloudFormation stack:

1. Open your Grafana Cloud portal.

2. Expand Observability > Cloud provider in the main menu.

3. Select AWS, the Configuration tab, and the CloudWatch metric streams card.

4. Click Launch CloudFormation stack.

   This opens a CloudFormation template in your AWS account in a new tab.

   Note

   The AWS account you are logged into is the account where resources are created. To use a different account, log out and log into the desired account before launching the CloudFormation stack.

5. Enter the access policy token with metric:write permissions you generated into the MetricsWriteToken field.

6. Update the FallbackS3BucketName if you already have an S3 bucket with a similar name, as bucket names must be globally unique.

7. (Optional) Add an AWS namespace you want your metric stream to filter metrics for. To specify additional configurations, such as more namespaces to include, either update the metric stream resource in AWS after CloudFormation creates it or use Terraform.

   If you leave this field blank, all AWS services that publish CloudWatch metrics are included.

1. Select the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox.

2. Click Create stack.

3. Wait for the stack creation to complete (usually 2-5 minutes).

4. After the stack creation is complete, navigate to the Outputs tab and copy the ARN from the AWS IAM role. You’ll need this ARN to create the resource metadata scrape job.

Configure optional settings

After the stack is created, you can configure optional settings to customize how metrics are sent to Grafana Cloud.

Set static labels

Static labels offer an additional way to filter and group your metrics in Grafana Cloud. You can set labels that apply to all metrics from this metric stream.

To set static labels:

1. Navigate to your Amazon Data Firehose in the AWS console.
2. Select the Configuration tab.
3. Select the Edit button for Destination settings.
4. Select the Add parameter button for Parameters - optional.
5. Enter a key-value pair in the corresponding text boxes.

Label requirements:

• Label keys must be prefixed with lbl_
• Label keys and values must be compatible with the Prometheus data model specification

Example:

• Key: lbl_environment
• Value: production

When querying in Grafana, do not include the lbl_ prefix, as in the following example query:

{job=~"cloud/aws/.+", environment="production"}

Set tag selection

Tag selection controls which AWS resource tags are attached to info metrics from the resource metadata scrape job. This helps lower cardinality by excluding dynamic tags that change frequently.

To set tag selection:

1. Navigate to your Amazon Data Firehose in the AWS console.
2. Select the Configuration tab.
3. Select the Edit button for Destination settings.
4. Select the Add parameter button for Parameters - optional.
5. Enter tag_selection as the parameter key.
6. Enter a comma-separated list of tags to include as the value.

Tag format requirements:

• Tags must begin with the prefix tag_
• Example: tag_Name,tag_Environment,tag_Owner

Special values:

• Empty string "": No tags are attached
• Parameter not set: All tags are attached (default behavior)

Disable the optional _average statistic

By default, the integration generates an _average statistic series from the CloudWatch Sum and SampleCount statistics to maintain compatibility with CloudWatch metric scrape jobs and dashboards.

For example, from aws_ec2_disk_read_bytes_sum and aws_ec2_disk_read_bytes_sample_count, the integration calculates aws_ec2_disk_read_bytes_average.

To disable this optional calculation:

1. Navigate to your Amazon Data Firehose in the AWS console.
2. Select the Configuration tab.
3. Select the Edit button for Destination settings.
4. Select the Add parameter button for Parameters - optional.
5. Enter include_average_statistic as the parameter key.
6. Enter false as the parameter value.

Create the resource metadata scrape job

Metrics from the metric stream only contain region and dimensions as labels. To enrich your metrics with additional metadata like ARNs and resource tags, create an AWS resource metadata scrape job.

Connect to AWS account

1. Open your Grafana Cloud portal.

2. Expand Observability > Cloud provider in the main menu.

3. Select AWS, the Configuration tab, and the CloudWatch metric streams card.

4. Under Continue setup for metrics streaming and create a resource metadata job, enter a name for your account (optional but recommended).

   Use a unique name with only alphanumeric characters, dashes, and underscores.

5. Paste the ARN you copied from the CloudFormation stack outputs.

6. Select the AWS regions where you have services you want to monitor.

   Tip

   Select the same regions where you created your metric stream.

7. Click Add account to verify the connection and save your account.

Add resource metadata job options

1. Enter a name for your resource metadata scrape job.

   Use a unique name with only alphanumeric characters, dashes, and underscores.

2. (Optional) Add static labels for easier filtering and grouping.

   These labels are added to all metrics exported by this scrape job.

3. Select the AWS services you want to scrape for resource metadata.

   Tip

   Choose the services you included in your metric stream. The resource metadata enriches the info metrics for these services.

4. (Optional) Click Edit next to a service to customize its settings:

   • Scrape interval: How frequently to fetch resource metadata (default: 5 minutes)
   • Tag filters: Limit which resources are scraped based on tag values

   Click Save service settings when finished.

5. Click Create job.

Verify your configuration

After completing the setup, verify that metrics are flowing correctly:

1. Check CloudWatch metric stream status:

   • Navigate to CloudWatch > Metric streams in the AWS console
   • Verify the metric stream shows Running state

2. Check Data Firehose delivery stream:

   • Navigate to Data Firehose in the AWS console
   • Verify the delivery stream shows Active status
   • Check that records are being delivered successfully

3. Verify metrics in Grafana Cloud:

   • Open Observability > Cloud Provider > AWS in your Grafana Cloud stack
   • Go to Services
   • You should see your AWS services listed with a Status of Sending data

4. Check for errors:

   • Verify the S3 fallback bucket is empty (no failed deliveries)
   • Check CloudWatch Logs for any Data Firehose errors

Troubleshooting

Metric stream not starting

Problem: The CloudWatch metric stream shows a failed or inactive state.

Solution:

• Verify the IAM role for the metric stream has permissions to write to Data Firehose
• Check that the Data Firehose delivery stream exists and is active
• Review CloudFormation events for error messages

No metrics appearing in Grafana

Problem: The metric stream is active, but no metrics appear in Grafana Cloud.

Solution:

• Verify the access policy token is correct and has metric:write permissions
• Check the Data Firehose delivery stream metrics for failed delivery attempts
• Review the S3 fallback bucket for failed batches
• Ensure the Grafana Cloud endpoint URL is correct in the Data Firehose configuration

High S3 storage costs

Problem: The S3 fallback bucket is accumulating data.

Solution:

• Failed batches indicate a delivery problem—check the Data Firehose error logs
• Verify the access policy token hasn’t expired
• Ensure the Grafana Cloud endpoint is reachable from your AWS region
• Consider adding S3 lifecycle policies to delete old failed batches