Grafana Cloud Monitor infrastructure Grafana integrations Integrations reference CloudWatch metrics

CloudWatch metrics integration for Grafana Cloud

The CloudWatch integration allows you to scrape AWS CloudWatch metrics without installing the Grafana agent. You can create multiple configurations called “scrape jobs” to separate concerns. Please note that we can only discover metrics for AWS resources that have tags applied to them. For more information, see the AWS docs.

The CloudWatch integration packages together many dashboards, a few of which reference Monitoring Artist.

CloudWatch integration vs CloudWatch data source

Grafana Cloud offers two solutions for visualizing your CloudWatch metrics, the integration or the data source. The data source allows you to keep your data in CloudWatch and build dashboards, rules, and alerts without pulling the data in to Grafana Cloud. The integration continuously pulls data from CloudWatch and pushes it to your Grafana Cloud Hosted Metrics Instance. This allows you to pull in your CloudWatch data from multiple AWS regions, query, and alert on it using the power of promql. The integration also gives you the ability to ingest the Tags from your AWS instance and make them available for querying/alerting.

This page only covers the integration which is managed via Connections in your Grafana Cloud instance. If you are not using the integration and are looking for documentation on the data source that can be found here, https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/.

Configuring the CloudWatch metrics integration for Grafana Cloud

1. In your Grafana Cloud stack, click Connections in the left-hand menu.
2. Find AWS and click its tile to open the AWS integration.
3. Find CloudWatch metrics and click its tile to open the integration
4. Click on the Configuration Details to begin the setup process

Creating scrape jobs

You can create a scrape job after successfully setting up a connection between Grafana Cloud and your AWS Account. This is done without transmitting secrets by setting up permission allow Grafana Cloud to assume a role in your AWS Account.

Setup AWS connection using CloudFormation

Scrape jobs can be named and connected to a specific AWS CloudWatch account. Each scrape job contains a number of services available to scrape. For example, you can create a job that scrapes metrics from your EC2 instances from a specific AWS account.

1. In the CloudWatch metrics tile, click Add scrape job.

2. Select Create Automatically in the first step of creating a new AWS role.

3. Follow the steps to create an IAM role for CloudFormation.

4. Return to the scrape job configuration UI

5. Paste the RoleARN output in the ARN field

6. Select regions where you have services you want to monitor

7. Click the Test Connection button to ensure the connection was setup successfully

Setup AWS connection using Terraform

Before you begin, make sure you have the Username / Instance ID for your Grafana Cloud Prometheus. This can be found by clicking on Details in the Prometheus card of the Cloud Portal.

You’ll find a Terraform snippet in this section that can be used to provision the IAM role needed to create the scrape jobs.

The input variables are:

• external_id: the Username / Instance ID for your Grafana Cloud Prometheus (see above for how to find it). An External ID is used by AWS to provide an extra layer of security when giving Grafana access to pull your CloudWatch metrics in to Grafana Cloud. To learn more about External IDs, see [why use an external ID?] (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html#external-id-purpose)

• iam_role_name: customizable name of the IAM role used by Grafana for the CloudWatch integration. The default value is GrafanaCloudWatchIntegration.

The output value is:

• role_arn: the IAM role ARN you need to use when creating the scrape job.

To run the Terraform file:

1. Configure the AWS CLI.

2. Copy this snippet into your Terraform file

   terraform

   terraform {
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 3.0"
       }
     }
   }
   locals {
     grafana_account_id = "008923505280"
   }
   variable "external_id" {
     type        = string
     description = "This is your Grafana Cloud identifier and is used for security purposes."
     validation {
       condition     = length(var.external_id) > 0
       error_message = "ExternalID is required."
     }
   }
   variable "iam_role_name" {
     type        = string
     default     = "GrafanaLabsCloudWatchIntegration"
     description = "Customize the name of the IAM role used by Grafana for the CloudWatch integration."
   }
   data "aws_iam_policy_document" "trust_grafana" {
     statement {
       effect = "Allow"
       principals {
         type        = "AWS"
         identifiers = ["arn:aws:iam::${local.grafana_account_id}:root"]
       }
       actions = ["sts:AssumeRole"]
       condition {
         test     = "StringEquals"
         variable = "sts:ExternalId"
         values   = [var.external_id]
       }
     }
   }
   resource "aws_iam_role" "grafana_labs_cloudwatch_integration" {
     name        = var.iam_role_name
     description = "Role used by Grafana CloudWatch integration."
     # Allow Grafana Labs' AWS account to assume this role.
     assume_role_policy = data.aws_iam_policy_document.trust_grafana.json
   
     # This policy allows the role to discover metrics via tags and export them.
     inline_policy {
       name = var.iam_role_name
       policy = jsonencode({
         Version = "2012-10-17"
         Statement = [
           {
             Effect = "Allow"
             Action = [
               "tag:GetResources",
               "cloudwatch:GetMetricData",
               "cloudwatch:ListMetrics",
               "apigateway:GET",
               "aps:ListWorkspaces",
               "autoscaling:DescribeAutoScalingGroups",
               "dms:DescribeReplicationInstances",
               "dms:DescribeReplicationTasks",
               "ec2:DescribeTransitGatewayAttachments",
               "ec2:DescribeSpotFleetRequests",
               "shield:ListProtections",
               "storagegateway:ListGateways",
               "storagegateway:ListTagsForResource"
             ]
             Resource = "*"
           }
         ]
       })
     }
   }
   output "role_arn" {
     value       = aws_iam_role.grafana_labs_cloudwatch_integration.arn
     description = "The ARN for the role created, copy this into Grafana Cloud installation."
   }

3. Run the terraform apply command in one of the following ways:

• By setting variables directly CLI

  bash

  terraform apply \
     -var="grafana_importer_external_id=<your external ID>" \
     -var="iam_role_name=GrafanaCloudWatchIntegration"

• Create a tfvars file
  
  <your-tfvars-file>.tfvars

  Add the following text:

  terraform

  grafana_importer_external_id="<your external ID>"
  iam_role_name="GrafanaCloudWatchIntegration"

  Run the following command:

  bash

  terraform apply -var-file="<your-tfvars-file>.tfvars"

1. Once the terraform apply command has finished creating the IAM Role, it will output your role_arn. For example:

   bash

   role_arn = "arn:aws:iam::<yourAWSAccountID>:role/<iam_role_name>"

2. Return to the scrape job configuration UI

3. Paste the role_arn terraform output in the ARN field

4. Select regions where you have services you want to monitor

5. Click the Test Connection button to ensure the connection was setup successfully

Setup AWS connection in the AWS Console

Please note that we recommend using automation as a best practice. Creating the role in the AWS IAM console requires many more steps.

Before you begin, make sure you have the Username / Instance ID for your Grafana Cloud Prometheus. This can be found by clicking on Details in the Prometheus card of the Cloud Portal.

1. Open the CloudWatch integration (configuration), click Add scrape job.

2. Select Manual and create a new role in your AWS IAM console.

3. Click the link to open the AWS IAM console and do the following:

4. In Roles, click Create role.

5. Choose AWS Account for Trusted entity type.

6. Choose Another AWS account.

7. In Account ID, enter the Grafana AWS account ID shown in the scrape job configuration.

8. Select Require external ID and enter the Username / Instance ID for your Grafana Cloud Prometheus (see above for how to find it). An External ID is used by AWS to provide an extra layer of security when giving Grafana access to pull your CloudWatch metrics in to Grafana Cloud. To learn more about External IDs, see [why use an external ID?] (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html#external-id-purpose)

9. Click Next: Permissions.

10. Click Create policy.

11. Go to the JSON section. Overwrite existing code with the code provided in the Grafana Cloud instructions.

12. At the bottom of each screen, click Next: Tags > Next: Review > Create policy.

13. Return to the scrape job configuration UI

14. Paste the ARN from your AWS IAM role in the ARN field

15. Select regions where you have services you want to monitor

16. Click the Test Connection button to ensure the connection was setup successfully

Creating your scrape job

After you have successfully tested the connection, create the scrape job.

1. Give your scrape job a unique name (containing only alphanumeric characters, dashes, and underscores).

2. Select a service to monitor. The default metrics and statistics are shown for each service.

3. Optionally, in the Statistics dropdown menu select what you want to gather from all metrics and click Apply to metrics selected. Alternatively, you can select the statistics for each metric. Note: Not all of the possible statistics are supported for every metric. Refer to AWS metric statistics for more information.

4. Optionally, deselect any metric you do not wish to include.

5. Select the scrape interval. (Currently, five minutes is supported.)

6. To add another scrape job, click +Service.

7. Optionally, select Include your AWS resource tags. Refer to Querying Tag Data for more information. Note: When you include tags, this will increase the total number of active series, which can have an impact on your Grafana Cloud Costs. Additionally, please ensure your tags adhere to AWS best practices (such as not containing personally identifiable information or other confidential or sensitive information).

8. Click Create scrape job to begin metrics collection.

9. Click Install to install our pre-build dashboards.

Scrape Job Resource Limit

When creating a job you might be presented with an error You reached the resource limit for a single scrape job. To continue, create several jobs with fewer services or regions.. There is a limit of 1024 resources per service in a job. This is intended to ensure metrics can be delivered to your Grafana Cloud instance in a timely manner.

The number of resources is determined by checking resources for each service per region. Let’s say you hit the limit setting up a job with name cloudwatch to cover EBS, EC2, and lambda in the us-east-1, us-east-2, eu-central-1, and eu-north-1 regions. You could create multiple jobs such as,

By Service

• cloudwatch-ebs: EBS in us-east-1, us-east-2, eu-central-1, and eu-north-1
• cloudwatch-ec2: EC2 in us-east-1, us-east-2, eu-central-1, and eu-north-1
• cloudwatch-lambda: lambda in us-east-1, us-east-2, eu-central-1, and eu-north-1

By Region

• cloudwatch-us-east: EBS, EC2, and lambda in us-east-1, and us-east-2
• cloudwatch-eu: EBS, EC2, and lambda in eu-central-1, and eu-north-1

By Service + Region

• cloudwatch-us-east-ebs: EBS in us-east-1, and us-east-2
• cloudwatch-eu-ebs: EBS in eu-central-1, and eu-north-1
• cloudwatch-us-east-ec2: EC2 in us-east-1, and us-east-2
• cloudwatch-eu-ec2: EC2 in eu-central-1, and eu-north-1
• cloudwatch-us-east-lambda: lambda in us-east-1, and us-east-2
• cloudwatch-eu-lambda: lambda in eu-central-1, and eu-north-1

Querying Tag Data

The CloudWatch metrics integration uses info metrics for all resources discovered while collecting data. If you chose to include tag data when setting up your Scrape Job, the info metrics will also include your tag data as labels with a name of tag_<tag_key> and value of <tag_value>. You can join these info metrics to service metrics by utilizing promql’s group_left function.

As an example if we produced the following metric data from ec2,

{__name__="aws_ec2_cpuutilization_maximum", account_id="asdf", dimension_InstanceId="i-asdf", name="arn:aws:ec2:us-east-2:asdf:instance/i-asdf"}
{__name__="aws_ec2_info", name="arn:aws:ec2:us-east-2:asdf:instance/i-asdf", tag_aws_autoscaling_groupName="eks-asdf", tag_aws_eks_cluster_name="cluster-asdf", tag_eks_nodegroup_name="ng-asdf"}

You can join these two metrics together with a query that looks like,

aws_ec2_cpuutilization_maximum + on (name) group_left(tag_aws_eks_cluster_name, tag_eks_nodegroup_name) aws_ec2_info

to produce an output of

{account_id="asdf", dimension_InstanceId="i-asdf", name="arn:aws:ec2:us-east-2:asdf:instance/i-asdf", tag_aws_eks_cluster_name="cluster-asdf", tag_eks_nodegroup_name="ng-asdf"}

You can adjust the collection of tags to join by changing the values in the group left. As an example, if you wanted to include tag_aws_autoscaling_groupName instead of tag_aws_eks_cluster_name and tag_eks_nodegroup_name you would use group_left(tag_aws_autoscaling_groupName)

Why are timestamps different between Grafana and CloudWatch/Why does my data look delayed?

The timestamp of a metric pulled by the CloudWatch integration is set to the time the metric is pulled. This might seem counter intuitive but in the end is intended to simplify writing alert queries.

The timestamps from the CloudWatch integration will always look more delayed than they actually are for a few reasons. These reasons are best illustrated via an example: let’s assume we are looking at a single metric, CPU Maximum, pulled every 5 minutes. This translates to the integration pulling data with a CloudWatch period of 5 minutes.

1. CloudWatch timestamps mark the beginning of a period not the end
   

2. CloudWatch samples are visible at the beginning of a period and aggregated through the period window
   

3. The CloudWatch integration pulls on a consistent interval and only requests data which has been fully aggregated
   

The end result is a Grafana Cloud timestamp of 0:08 for a metric CloudWatch has stamped at 0:00. If the CloudWatch timestamp was used instead, metrics would appear to be 8 minutes old when ingested and any alert queries written would need to consider this extra variable delay. The choice to use the pull timestamp gives the appearance of an 8 minute delay. But in reality only 3 minutes has passed since the value stopped being updated and alert queries can remain simple.

Services

The CloudWatch integration supports the following services and allows you to pick from a wide array of available metrics and statistics. Metrics and statistics which are bolded are included in our default configuration

AWS/ACMPrivateCA

• Scrape Intervals: 5 minutes

AWS/AmazonMQ

• Scrape Intervals: 5 minutes

AWS/ApiGateway

• Scrape Intervals: 5 minutes

AWS/AppStream

• Scrape Intervals: 5 minutes

AWS/AppSync

• Scrape Intervals: 5 minutes

AWS/ApplicationELB

• Scrape Intervals: 5 minutes

AWS/Athena

• Scrape Intervals: 5 minutes

AWS/AutoScaling

• Scrape Intervals: 5 minutes

AWS/Backup

• Scrape Intervals: 5 minutes

AWS/Billing

Note this service only produces metrics to specific regions in AWS. Any jobs configured with this service will only gather data from the us-east-1 region(s)

• Scrape Intervals: 5 minutes

AWS/Cassandra

• Scrape Intervals: 5 minutes

AWS/CertificateManager

• Scrape Intervals: 5 minutes

AWS/CloudFront

• Scrape Intervals: 5 minutes

AWS/Cognito

• Scrape Intervals: 5 minutes

AWS/DDoSProtection

• Scrape Intervals: 5 minutes

AWS/DMS

• Scrape Intervals: 5 minutes

AWS/DX

• Scrape Intervals: 5 minutes

AWS/DocDB

• Scrape Intervals: 5 minutes

AWS/DynamoDB

• Scrape Intervals: 5 minutes

AWS/EBS

• Scrape Intervals: 5 minutes

AWS/EC2

• Scrape Intervals: 5 minutes

AWS/EC2Spot

• Scrape Intervals: 5 minutes

AWS/ECS

• Scrape Intervals: 5 minutes

AWS/EFS

• Scrape Intervals: 5 minutes

AWS/ELB

• Scrape Intervals: 5 minutes

AWS/ES

• Scrape Intervals: 5 minutes

AWS/ElastiCache

• Scrape Intervals: 5 minutes

AWS/ElasticBeanstalk

• Scrape Intervals: 5 minutes

AWS/ElasticMapReduce

• Scrape Intervals: 5 minutes

AWS/FSx

• Scrape Intervals: 5 minutes

AWS/Firehose

• Scrape Intervals: 5 minutes

AWS/GameLift

• Scrape Intervals: 5 minutes

AWS/GlobalAccelerator

• Scrape Intervals: 5 minutes

AWS/IoT

• Scrape Intervals: 5 minutes

AWS/Kafka

• Scrape Intervals: 5 minutes

AWS/Kinesis

• Scrape Intervals: 5 minutes

AWS/KinesisAnalytics

• Scrape Intervals: 5 minutes

AWS/Lambda

• Scrape Intervals: 5 minutes

AWS/MediaConnect

• Scrape Intervals: 5 minutes

AWS/MediaTailor

• Scrape Intervals: 5 minutes

AWS/NATGateway

• Scrape Intervals: 5 minutes

AWS/Neptune

• Scrape Intervals: 5 minutes

AWS/NetworkELB

• Scrape Intervals: 5 minutes

AWS/NetworkFirewall

• Scrape Intervals: 5 minutes

AWS/PrivateLinkEndpoints

• Scrape Intervals: 5 minutes

AWS/PrivateLinkServices

• Scrape Intervals: 5 minutes

AWS/Prometheus

• Scrape Intervals: 5 minutes

AWS/RDS

• Scrape Intervals: 5 minutes

AWS/Redshift

• Scrape Intervals: 5 minutes

AWS/Route53

• Scrape Intervals: 5 minutes

AWS/Route53Resolver

• Scrape Intervals: 5 minutes

AWS/S3

• Scrape Intervals: 5 minutes

AWS/SES

• Scrape Intervals: 5 minutes

AWS/SNS

• Scrape Intervals: 5 minutes

AWS/SQS

• Scrape Intervals: 5 minutes