Amazon CloudWatch data source on Grafana Labs

Configure CloudWatch

Tue, 18 Aug 2026 10:15:08 +0000

Configure the Amazon CloudWatch data source

This document provides instructions for configuring the Amazon CloudWatch data source and explains available configuration options. For general information on adding and managing data sources, refer to Data source management.

Before you begin

You must have the Organization administrator role to configure the CloudWatch data source. Organization administrators can also configure the data source via YAML with the Grafana provisioning system.
Grafana comes with a built-in CloudWatch data source plugin, so you do not need to install a plugin.
Familiarize yourself with your CloudWatch security configuration and gather any necessary security certificates, client certificates, and client keys.

Key concepts

If you’re new to AWS, these terms are used throughout the configuration:

Term	Description
IAM policy	A JSON document attached to an identity that grants permissions to AWS API actions.
IAM role	An identity with permissions that trusted entities can assume temporarily, instead of using long-lived keys.
Assume role	An AWS mechanism that lets one identity temporarily take on the permissions of an IAM role, often used for cross-account access.
External ID	A shared value in a role’s trust policy that prevents another party from assuming the role on your behalf.
STS (Security Token Service)	The AWS service that issues the short-lived temporary credentials used when assuming a role.
Cross-account observability (OAM)	An AWS feature, managed through Observability Access Manager, that links monitoring and source accounts so you can query metrics and logs across accounts.

Add the CloudWatch data source

Complete the following steps to set up a new CloudWatch data source:

Click Connections in the left-side menu.
Click Add new connection
Type CloudWatch in the search bar.
Select the CloudWatch data source.
Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you will set up your CloudWatch configuration.

Configure the data source in the UI

The following are configuration options for the CloudWatch data source.

Setting	Description
Name	The data source name. Sets the name you use to refer to the data source in panels and queries.
Default	Toggle to select as the default name in dashboard panels. When you go to a dashboard panel, this will be the default selected data source.

Grafana plugin requests to AWS are made on behalf of an AWS Identity and Access Management (IAM) role or IAM user. The IAM user or IAM role must have the associated policies to perform certain API actions.

For authentication options and configuration details, refer to AWS authentication.

Setting	Description
Authentication Provider	Specify which AWS credentials chain to use. A Grafana plugin’s requests to AWS are made on behalf of an IAM role or IAM user. The IAM user or IAM role must have the necessary policies to perform the required API actions.

Access & secret key:

You must use both an access key ID and a secret access key to authenticate.

Setting	Description
Access Key ID	Enter your key ID.
Secret Access Key	Enter the secret access key.

Credentials file:

When you select the Credentials file authentication provider, you can specify which profile to read from the AWS shared credentials file.

Setting	Description
Credentials Profile Name	The profile name in `~/.aws/credentials`, as specified in the credentials file. Leave blank to use the `default` profile.

Assume Role:

Setting	Description
Assume Role ARN	Optional. Specify the ARN of an IAM role to assume. This ensures the selected authentication method is used to assume the role, not used directly.
External ID	If you’re assuming a role in another AWS account that requires an external ID, specify it here.

Additional Settings:

Setting	Description
Endpoint	Optional. Specify a custom endpoint for the AWS service.
Default Region	Specify the AWS region. Example: If the region is US West (Oregon), use `us-west-2`.
Namespaces of Custom Metrics	Add one or more custom metric namespaces, separated by commas (for example,`Namespace1,Namespace2`). Grafana can’t automatically load custom namespaces using the CloudWatch GetMetricData API. To make custom metrics available in the query editor, manually specify the namespaces in the `Namespaces of Custom Metrics` field in the data source configuration.

Proxy configuration:

The Proxy Configuration settings appear only when per-data source HTTP proxy support is enabled with the per_datasource_http_proxy_enabled option in the Grafana configuration file. They let each CloudWatch data source use its own outbound HTTP proxy for AWS requests.

Setting	Description
Proxy Type	Select how the proxy is configured: `Environment (default)` uses the `HTTP_PROXY` and `HTTPS_PROXY` environment variables, `None` disables the proxy, and `URL` lets you specify a proxy URL. Don’t set this when Secure Socks Proxy is enabled.
Proxy URL	Only when Proxy Type is `URL`. The proxy URL, for example `https://localhost:3004`. Don’t include the username or password in the URL.
Proxy Username	Only when Proxy Type is `URL`. Optional. The proxy username.
Proxy Password	Only when Proxy Type is `URL`. Optional. The proxy password.

CloudWatch Logs:

Setting	Description
Query Result Timeout	Grafana polls CloudWatch Logs every second until AWS returns a `Done` status or the timeout is reached. An error is returned if the timeout is exceeded. For alerting, the timeout defined in the Grafana configuration file takes precedence. Enter a valid duration string, such as `30m`, `30s` or `200ms`. The default is `30m`.
Default Log Groups	Optional. Specify the default log groups for CloudWatch Logs queries.

Application Signals trace link:

Setting	Description
Data source	Select the Application Signals data source from the drop-down menu.

Grafana automatically creates a link to a trace in Application Signals data source if logs contain the @xrayTraceId field. To use this feature, you must already have an Application Signals data source configured. For details, refer to the Application Signals data source docs. To view the Application Signals link, select the log row in either the Explore view or dashboard Logs panel to view the log details section.

To log the @xrayTraceId, refer to the AWS Application Signals documentation. To provide the field to Grafana, your log queries must also contain the @xrayTraceId field, for example by using the query fields @message, @xrayTraceId.

Private data source connect - Only for Grafana Cloud users.

Setting	Description
Private data source connect	Establishes a private, secured connection between a Grafana Cloud stack and data sources within a private network. Use the drop-down to locate the PDC URL. For setup instructions, refer to Private data source connect (PDC) and Configure PDC. Click Manage private data source connect to open your PDC connection page and view your configuration details.

After configuring your Amazon CloudWatch data source options, click Save & test at the bottom to test the connection. When the test succeeds, Grafana confirms that it reached both CloudWatch APIs:

1. Successfully queried the CloudWatch metrics API.
2. Successfully queried the CloudWatch logs API.

Note
To troubleshoot issues while setting up the CloudWatch data source, check the /var/log/grafana/grafana.log file. Common issues include invalid credentials, missing regions and metrics-only credentials.

IAM policy examples

To read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana permissions via IAM. You can attach these permissions to the IAM role or IAM user you configured in AWS authentication.

Metrics-only permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    }
  ]
}

Logs-only permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:ListAggregateLogGroupSummaries",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Metrics and logs permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "CloudWatch:DescribeAlarmsForMetric",
        "CloudWatch:DescribeAlarmHistory",
        "CloudWatch:DescribeAlarms",
        "CloudWatch:ListMetrics",
        "CloudWatch:GetMetricData",
        "CloudWatch:GetInsightRuleReport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourceMetricsFromPerformanceInsights",
      "Effect": "Allow",
      "Action": "pi:GetResourceMetrics",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingLogsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:ListAggregateLogGroupSummaries",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
      "Effect": "Allow",
      "Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Cross-account observability permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Note
Cross-account observability lets you retrieve metrics and logs across different accounts in a single region, but you can’t query EC2 Instance Attributes across accounts because those come from the EC2 API and not the CloudWatch API.

For more information on configuring authentication, refer to Configure AWS authentication.

Permissions reference

The following table explains why each permission is needed and whether it’s required, so you can scope a policy to only the features you use. Grant cloudwatch:GetMetricData only for metrics, the logs: actions only for logs, and the EC2 and tag actions only if you use the related template variables. Granting a permission that you don’t use has no effect on functionality.

Permission	Purpose	Required
`cloudwatch:ListMetrics`	Lists available metrics and populates the namespace, metric, and dimension fields in the query editor.	Required for metrics queries and metrics template variables.
`cloudwatch:GetMetricData`	Retrieves metric data points for queries and metric math expressions.	Required for metrics queries.
`cloudwatch:GetInsightRuleReport`	Retrieves data from CloudWatch Contributor Insights rules.	Optional. Only needed if you query Contributor Insights rules.
`cloudwatch:DescribeAlarms`, `cloudwatch:DescribeAlarmsForMetric`, `cloudwatch:DescribeAlarmHistory`	Reads CloudWatch alarm configuration and history.	Optional. Only needed if you display alarm data.
`pi:GetResourceMetrics`	Retrieves Performance Insights metrics for supported resources, such as RDS databases.	Optional. Only needed if you query Performance Insights metrics.
`logs:DescribeLogGroups`	Lists log groups and populates the log group selector.	Required for logs queries and the Log Groups template variable.
`logs:StartQuery`, `logs:StopQuery`, `logs:GetQueryResults`	Runs CloudWatch Logs Insights queries and retrieves their results.	Required for logs queries.
`logs:GetLogGroupFields`	Retrieves the fields available in a log group for autocomplete.	Optional. Improves the logs query editor experience.
`logs:GetLogEvents`	Retrieves individual log events.	Optional. Only needed for queries that read raw log events.
`logs:Unmask`	Reveals data masked by a CloudWatch Logs data protection policy.	Optional. Only needed to view masked sensitive data.
`ec2:DescribeRegions`	Lists available AWS regions.	Optional. Only needed if you rely on dynamic region discovery.
`ec2:DescribeInstances`, `ec2:DescribeTags`	Resolves EC2 instance attributes and tags for dimension and EC2 template variables.	Optional. Only needed for EC2 Instance Attributes template variables.
`ec2:DescribeVolumes`	Resolves EBS volume IDs for the EBS Volume IDs template variable.	Optional. Only needed for EBS Volume IDs template variables.
`tag:GetResources`	Resolves resource ARNs for the Resource ARNs template variable.	Optional. Only needed for Resource ARNs template variables.
`oam:ListSinks`, `oam:ListAttachedLinks`	Discovers linked accounts for CloudWatch cross-account observability.	Optional. Only needed for cross-account observability.

When you use an Assume Role ARN, attach these query permissions to the assumed role. The primary credentials only need permission to perform sts:AssumeRole. For details, refer to Assume a role.

CloudWatch Logs data protection

CloudWatch Logs can protect data by applying log group data protection policies. When data protection is enabled for a log group, any sensitive data that matches the identifiers you select is automatically masked. To view masked data, your IAM role or user must have the logs:Unmask permission. For more details, refer to the AWS guide on masking sensitive log data.

Configure the data source with grafana.ini

The Grafana configuration file includes an AWS section where you can configure data source options:

Configuration option	Description
`allowed_auth_providers`	Specifies which authentication providers are allowed for the CloudWatch data source as a comma-separated list. The default is `default,keys,credentials`: `default` (AWS SDK Default), `keys` (Access and secret key), and `credentials` (Credentials file). The `ec2_iam_role` (EC2 IAM role) provider is also available but isn’t enabled by default.
`assume_role_enabled`	Allows you to disable `assume role (ARN)` in the CloudWatch data source. The assume role (ARN) is enabled by default in open source Grafana.
`per_datasource_http_proxy_enabled`	Allows each CloudWatch data source instance to use its own HTTP proxy configuration for requests to AWS, instead of a shared proxy. Disabled by default. Set to `true` to enable.
`list_metrics_page_limit`	Sets the limit of List Metrics API pages. When a custom namespace is specified in the query editor, the List Metrics API populates the Metrics field and Dimension fields. The API is paginated and returns up to 500 results per page, and the data source also limits the number of pages to 500 by default. This setting customizes that limit.

Provision the data source

You can define and configure the data source in YAML files as part of the Grafana provisioning system. For more information about provisioning and available configuration options, refer to Provision Grafana.

Using AWS SDK (default):

apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      defaultRegion: eu-west-2

Using credentials’ profile name (non-default):

apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: credentials
      defaultRegion: eu-west-2
      customMetricsNamespaces: 'CWAgent,CustomNameSpace'
      profile: secondary

Using accessKey and secretKey:

apiVersion: 1

datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: keys
      defaultRegion: eu-west-2
    secureJsonData:
      accessKey: '<your access key>'
      secretKey: '<your secret key>'

Using AWS SDK Default and ARN of IAM Role to Assume:

apiVersion: 1
datasources:
  - name: CloudWatch
    type: CloudWatch
    jsonData:
      authType: default
      assumeRoleArn: arn:aws:iam::123456789012:root
      defaultRegion: eu-west-2

Configure with Terraform

You can configure the CloudWatch data source using Terraform with the Grafana Terraform provider.

For more information about provisioning resources with Terraform, refer to Grafana as code using Terraform.

The following example uses the Access and secret key authentication method:

resource "grafana_data_source" "cloudwatch" {
  name = "CloudWatch"
  type = "cloudwatch"

  json_data_encoded = jsonencode({
    authType      = "keys"
    defaultRegion = "eu-west-2"
  })

  secure_json_data_encoded = jsonencode({
    accessKey = "<YOUR_ACCESS_KEY>"
    secretKey = "<YOUR_SECRET_KEY>"
  })
}

The following example uses the AWS SDK Default authentication method with an assumed role:

resource "grafana_data_source" "cloudwatch" {
  name = "CloudWatch"
  type = "cloudwatch"

  json_data_encoded = jsonencode({
    authType      = "default"
    assumeRoleArn = "arn:aws:iam::123456789012:role/<ROLE_NAME>"
    defaultRegion = "eu-west-2"
  })
}

Replace the placeholders with your own values:

<YOUR_ACCESS_KEY> and <YOUR_SECRET_KEY>: The AWS access key ID and secret access key for the Access and secret key method.
<ROLE_NAME>: The name of the IAM role to assume.

For all available configuration options, refer to the Grafana provider data source resource documentation.

Configure AWS authentication

Tue, 18 Aug 2026 10:15:08 +0000

Configure AWS authentication

Grafana data source plugins make requests to AWS on behalf of an AWS Identity and Access Management (IAM) role or IAM user. The IAM user or IAM role must have the associated policies to perform certain API actions for querying data source data. Since these policies are specific to each data source, refer to individual data source plugin documentation for details.

The Grafana backend performs all AWS API requests server-side using the official AWS SDK.

This document explores the following topics:

Select an authentication method

Available authentication methods depend on your configuration and the environment where Grafana runs.

Open source Grafana enables the AWS SDK Default, Credentials file, and Access & secret key methods by default. Cloud Grafana enables Access & secret key and Grafana Assume Role by default. Users with server configuration access can enable or disable specific auth providers as needed. For more information, refer to the allowed_auth_providers documentation.

AWS SDK Default uses the default provider from the AWS SDK for Go without custom configuration. This method requires configuring AWS credentials outside Grafana through the CLI, or by attaching credentials directly to an EC2 instance, in an ECS task, or for a Service Account in a Kubernetes cluster. You can attach permissions directly to the data source with AWS SDK Default or combine it with the optional Assume Role ARN field.
Credentials file maps to the SharedCredentialsProvider provider in the AWS SDK for Go. This method reads the AWS shared credentials file for a specified profile. Unlike AWS SDK Default which also reads the shared credentials file, this option lets you specify a profile directly without environment variables. This option provides no fallback to other credential providers and fails if the file credentials are invalid.
Access & secret key corresponds to the StaticProvider and authenticates using a specified access key ID and secret key pair. This method doesn’t provide fallback authentication and fails if the key pair is invalid. It’s available in Grafana Cloud, though Grafana Assume Role is recommended there because it avoids managing long-term access keys.
Grafana Assume Role - With this authentication method, Grafana Cloud users create an AWS IAM role that has a trust relationship with the Grafana AWS account. Grafana uses STS to generate temporary credentials on its behalf. This method eliminates the need to generate secret and access keys for users. Refer to Use Grafana Assume Role for more information.
Workspace IAM Role corresponds to the EC2RoleProvider. The EC2RoleProvider retrieves credentials from a role attached to the EC2 instance running Grafana. While AWS SDK Default can achieve similar results, this option provides no fallback authentication. Amazon Managed Grafana enables this option by default.

Assume a role

Specify an IAM role to assume in the Assume Role ARN field.

When you configure Assume Role ARN, Grafana uses the provided credentials to perform an sts:AssumeRole call. The primary authentication method only needs permission to assume the role, while the assumed role requires CloudWatch access permissions.

For example, you might use one set of long-term credentials across all AWS data sources but want to limit each data source’s AWS access (such as separating staging and production data access). You could create separate credentials for each data source with distinct permissions, but this approach requires managing and rotating multiple secret and access keys across many data source instances.

Instead, assume role functionality lets you use one set of AWS credentials across all AWS data sources with a single permission: the ability to assume roles through STS. You then create separate IAM roles for each data source that specify temporary permissions. Since IAM roles are not credentials, they require no rotation and simplify management.

If the Assume Role ARN field is left empty, Grafana uses the provided credentials from the selected authentication method directly, and permissions to AWS data must be attached directly to those credentials. The Assume Role ARN field is optional for all authentication methods except for Grafana Assume Role.

To disable this feature in open source Grafana or Grafana Enterprise, refer to assume_role_enabled.

Use an external ID

Note
You cannot use an external ID for the Grafana Assume Role authentication provider.

To assume a role in another account created with an external ID, specify the external ID in the External ID field.

For more information, refer to the AWS documentation on external ID.

Use a custom endpoint

Use the Endpoint field to specify a custom endpoint URL that overrides the default AWS service API endpoint. Leave this field blank to use the default generated endpoint.

For more information about using service endpoints, refer to the AWS service endpoints documentation.

Use an AWS credentials file

Create a file at ~/.aws/credentials, the HOME path for the user running the grafana-server service.

Note
If the credentials file appears to be in the correct location but isn’t working, move your aws file to /usr/share/grafana/ and set the credentials file permissions to 0644.

Credentials file example:

[default]
aws_access_key_id = asdsadasdasdasd
aws_secret_access_key = dasdasdsadasdasdasdsa
region = us-west-2

Use EKS IAM roles for service accounts

EKS IAM roles for service accounts (IRSA) are an AWS EKS feature that allows pods to assume IAM roles without storing long-term credentials. When you configure IRSA in your EKS cluster, AWS injects temporary credentials into your pod as projected volume mounts.

In Grafana containers, the process runs as user 472 (grafana). By default, Kubernetes mounts the projected credentials with permissions for the root user only.

To grant user 472 permission to access the credentials, and prevent fallback to the IAM role attached to the EC2 instance, set a security context for your pod.

Security context example:

securityContext:
  fsGroup: 472
  runAsUser: 472
  runAsGroup: 472

Use Grafana Assume Role

Note
Grafana Assume Role is available only in Grafana Cloud. It’s currently supported only for the Amazon CloudWatch and Amazon Athena data sources.

The Grafana Assume Role authentication provider lets you access AWS without creating or managing long-term AWS IAM users or rotating access keys. Instead, you can create an IAM role that has permissions to access CloudWatch and trusts a Grafana AWS account.

The Grafana AWS account then makes a Security Token Service (STS) request to generate temporary credentials for your AWS data. This request includes an externalID unique to your Grafana Cloud account, which ensures users can access only their own AWS resources.

Note
The trust policy names the shared Grafana AWS account as the trusted principal, but this doesn’t grant every Grafana customer access to your role. The sts:ExternalId condition restricts the trust to your unique external ID, so AWS only allows an assume-role request that presents that exact value. Grafana sends your external ID only for your own data source requests. The external ID isn’t a secret, but combined with the account-level trust it ensures one customer can’t assume another customer’s role. Keep the sts:ExternalId condition in the trust policy — removing it would let any account that Grafana trusts assume your role.

For more information, refer to the AWS documentation on external ID.

To use the Grafana Assume Role:

Create a new CloudWatch data source (or update an existing one) and select Grafana Assume Role as an authentication provider.
In the AWS Console, create a new IAM role, and under Trusted entity type, select Another AWS account as the trusted Entity.
Enter the Grafana account id (displayed in the instructions box on the Settings tab of the CloudWatch data source configuration) and check the Require external ID box.
Enter the external ID specified in the instructions box on the Settings tab of the CloudWatch data source configuration in Grafana. This external ID will be unique to your Grafana instance.
Attach any required permissions you would like Grafana to be able to access on your behalf (for example, CloudWatch Logs and CloudWatch Metrics policies).
Give the role a name and description, and click Create role.
Copy the ARN of the role you just created and paste it into the Assume Role ARN field on the Settings tab of CloudWatch data source configuration in Grafana.

Sample Trust Relationship for an IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": {Grafana's AWS Account}
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": {External ID unique to your account}
                }
            }
        }
    ]
}

Amazon CloudWatch query editor

Tue, 18 Aug 2026 10:15:08 +0000

Amazon CloudWatch query editor

Grafana provides a query editor for the CloudWatch data source, which allows you to query, visualize, and alert on logs and metrics stored in Amazon CloudWatch. It is located on the Explore page. For general documentation on querying data sources in Grafana, refer to Query and transform data.

Key concepts

If you’re new to CloudWatch, these terms are used throughout this documentation:

Term	Description
Namespace	A container for CloudWatch metrics, usually one per AWS service, such as `AWS/EC2` or `AWS/Lambda`.
Metric	A time-ordered set of data points published to CloudWatch, such as `CPUUtilization`.
Dimension	A name-value pair that identifies a specific resource for a metric, such as `InstanceId=i-1234567890abcdef0`.
Statistic	An aggregation applied to metric data over a period, such as `Average`, `Sum`, or `Maximum`.
Period	The length of time, in seconds, used to aggregate metric data points.
Metric math	Expressions that combine or transform metrics, such as computing a rate or a sum across metrics.
Metrics Insights	A SQL-like query engine for filtering and aggregating metrics in near real time.
Search expression	A CloudWatch expression that matches multiple metrics at query time, often used with wildcards.
Logs Insights QL	The CloudWatch Logs query language used to search and aggregate log data.

Choose a query editing mode

The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor.

Select the API to query using the drop-down to the right of the Region setting.

CloudWatch Metrics query editor components

The following are the components of the CloudWatch query editor.

Setting	Description
Region	Select an AWS region if it differs from the default.
Namespace	The AWS service namespace. Examples: `AWS/EC2`, `AWS/Lambda`.
Metric name	The name of the metric you want to visualize. Example: `CPUUtilization`.
Statistic	Choose how to aggregate your data. Options are `Average`, `Maximum`, `Minimum`, `Sum`, `SampleCount`, and `IQM`. You can also enter a custom value, such as a percentile like `p99`.
Dimensions	Select dimensions from the drop-down. Examples: `InstanceId`, `FunctionName`, `latency`. You can add several dimensions to your query.
Match exact	Optional. When enabled, this option restricts query results to metrics that precisely match the specified dimensions and their values. All dimensions of the queried metric must be explicitly defined in the query to ensure an exact schema match. If disabled, the query will also return metrics that match the defined schema but possess additional dimensions.
ID	Optional. Unique identifier required by the GetMetricData API for referencing queries in math expressions. Must start with a lowercase letter and can include letters, numbers, and underscores. If not specified, Grafana generates an ID using the pattern `query[refId]` (for example, `queryA`for the first query row).
Period	The minimum time interval, in seconds, between data points. The default is `auto`. CloudWatch accepts periods of 1, 5, 10, 30, or any multiple of 60 seconds. When set to `auto` or left blank, Grafana calculates the period by dividing the time range in seconds by 2000, then rounding up to the next value among 60, 300, 900, 3600, 21600, and 86400. The available periods depend on the age of the data: because CloudWatch stores older metrics at a lower resolution under its retention policy, the shorter periods aren’t available for older time ranges.
Label	Optional. Add a customized time series legend name. The label field overrides the default metric legend name using CloudWatch dynamic labels. Time-based dynamic labels like ${MIN_MAX_TIME_RANGE} derive legend values from the current timezone in the time range picker. For the full list of label patterns and limitations, refer to CloudWatch dynamic labels.

Use Builder and Code modes

Both the Metric Search and Metric Insights query types provide two editing modes. Use the Builder and Code toggle in the query editor to switch between them.

Builder mode

Builder mode lets you construct a query from drop-downs and form fields without writing any syntax:

For Metric Search queries, select the namespace, metric name, statistic, and dimensions.
For Metric Insights queries, select the namespace, metric name, filter, group, and order options using the Metrics Insights keywords. Grafana constructs a SQL query based on your selections.

Code mode

Code mode lets you write your query directly in a code editor:

For Metric Search queries, write a metric math or SEARCH expression.
For Metric Insights queries, write a Metrics Insights SQL query.

The code editor includes a built-in autocomplete feature that suggests keywords, aggregations, namespaces, metrics, labels, and label values. Suggestions appear after typing a space, comma, or dollar ($) character, or by pressing CTRL+Space.

Note
Template variables in the code editor can interfere with autocompletion.

To run the query, click Run query above the code editor.

Query CloudWatch metrics

You can create two types of queries in the CloudWatch query editor:

Metric Search, which helps you retrieve and filter available metrics.
Metric Insights, which uses the Metrics Insights feature to fetch time series data.

The query type you use depends on how you want to interact with AWS metrics. Use the drop-down in the upper middle of the query editor to select which type you want to create.

Metric Search queries

Metric search queries help you discover and filter available metrics. These queries use wildcards and filters to find metrics without needing to know exact metric names.

A valid metric query requires a specified namespace, metric name, and at least one statistic. Dimensions are optional, but if included, you must provide both a key and a value.

The Match Exact option controls how dimension filtering is applied to metric queries. When you enable match exact, the query returns only metrics whose dimensions precisely match the specified criteria.

This requires the following:

All dimensions present on the target metric must be explicitly specified.
Dimensions you don’t want to filter by must use a wildcard (*) filter.
The metric schema must match exactly as defined in the CloudWatch metric schema documentation.

When Match Exact is disabled, you can specify any subset of dimensions for filtering. The query returns metrics that:

Match the specified namespace and metric name.
Match all defined dimension filters.
May contain additional dimensions beyond those specified.

This mode provides more flexible querying but may return metrics with unexpected additional dimensions.

Because the data source retrieves metrics through the CloudWatch GetMetricData API, a single query returns data for up to 100 metrics matching your filter criteria, which is the per-request limit that AWS enforces.

Enhance metric queries using template variables to create dynamic, reusable dashboards.

Create dynamic queries with dimension wildcards

Use the asterisk (*) wildcard for dimension values to create dynamic queries that automatically monitor changing sets of AWS resources.

The query returns the average CPU utilization for all EC2 instances in the default region. With Match Exact disabled and InstanceId using a wildcard, the query retrieves metrics for any EC2 instance regardless of additional dimensions.

Auto-scaling events add new instances to the graph without manual instance ID tracking. As with other metric search queries, AWS returns up to 100 metrics per GetMetricData request.

Click the Query inspector button and select Meta Data to see the search expression that’s automatically built to support wildcards.

To learn more about search expressions, refer to the CloudWatch documentation. The search expression is defined by default in such a way that the queried metrics must match the defined dimension names exactly. This means that in the example, the query returns only metrics with exactly one dimension containing the name InstanceId.

Disabling Match Exact includes metrics with additional dimensions and creates a search expression even without wildcards. Grafana searches for any metric matching at least the namespace, metric name, and all defined dimensions.

Use multi-value template variables

When defining dimension values based on multi-valued template variables, the data source uses a search expression to query for the matching metrics. This enables the use of multiple template variables in one query, and also lets you use template variables for queries that have the Match Exact option disabled.

Search expressions are limited to 1,024 characters, so your query might fail if you have a long list of values. We recommend using the asterisk (*) wildcard instead of the All option to query all metrics that have any value for a certain dimension name.

Multi-valued template variables are supported only for dimension values. Using multi-valued template variables for Region, Namespace, or Metric Name is not supported.

Use metric math expressions

Create new time series metrics using mathematical functions on CloudWatch metrics. This supports arithmetic operators, unary subtraction, and other functions. For available functions, refer to AWS Metric Math.

To apply arithmetic operations, assign a unique string ID to the raw metric, then reference this ID in the Expression field of the new metric.

Note
If you use the expression field to reference another query, such as queryA * 2, you can’t create an alert rule based on that query.

Query metrics across AWS monitoring accounts

When you select Builder mode within the Metric search editor, a new Account field is displayed. Use the Account field to specify which of the linked monitoring accounts to target for the given query. By default, the All option is specified, which will target all linked accounts.

While in Code mode, you can specify any math expression. If the Monitoring account badge displays in the query editor header, all SEARCH expressions entered in this field will be cross-account by default and can query metrics from linked accounts. Note that while queries run cross-account, the autocomplete feature currently doesn’t fetch cross-account resources, so you’ll need to manually specify resource names when writing cross-account queries. You can limit the search to one or a set of accounts, as documented in the AWS documentation.

Period macro

If you use a CloudWatch SEARCH expression, consider using the $__period_auto macro rather than specifying a period explicitly. The $__period_auto macro will resolve to a CloudWatch period that is suitable for the chosen time range.

Deep-link Grafana panels to the CloudWatch console

Left-clicking a time series in the panel displays a context menu with a link to View in CloudWatch console. Clicking the link opens a new tab that takes you to the CloudWatch console and displays all metrics for that query. If you’re not logged in to the CloudWatch console, the link forwards you to the login page. The link provided is valid for any account but displays the expected metrics only if you’re logged in to the account that corresponds to the selected data source in Grafana.

This feature is not available for metrics based on metric math expressions.

Use Metric Insights syntax

Metric Insights uses a dialect of SQL and this query syntax:

SELECT FUNCTION(MetricName)
FROM Namespace | SCHEMA(...)
[ WHERE labelKey OPERATOR labelValue [AND|...]]
[ GROUP BY labelKey [, ...]]
[ ORDER BY FUNCTION() [DESC | ASC] ]
[ LIMIT number]

For details about the Metrics Insights syntax, refer to the AWS reference documentation.

For information about Metrics Insights limits, refer to the AWS feature documentation.

You can also augment queries by using template variables.

Use Metrics Insights keywords

This table summarizes common Metrics Insights query keywords:

Keyword	Description
`FUNCTION`	Required. Specifies the aggregate function to use, and also specifies the name of the metric to be queried. Valid values are AVG, COUNT, MAX, MIN, and SUM.
`MetricName`	Required. For example, `CPUUtilization`.
`FROM`	Required. Specifies the metric’s source. You can specify either the metric namespace that contains the metric to be queried, or a SCHEMA table function. Namespace examples include `AWS/EC2`, `AWS/Lambda`.
`SCHEMA`	Optional. Narrows the query results to only the metrics that are an exact match, or to metrics that do not match.
`WHERE`	Optional. Filters the query results to only the metrics that match your specified expression. For example, `WHERE InstanceType != 'c3.4xlarge'`.
`GROUP BY`	Optional. Groups the query results into multiple time series. For example, `GROUP BY ServiceName`.
`ORDER BY`	Optional. Specifies the order in which time series are returned. Options are `ASC`, `DESC`.
`LIMIT`	Optional. Limits the number of time series returned.

Query CloudWatch Logs

The logs query editor helps you write CloudWatch Logs queries across a selected region and a set of log groups or Amazon CloudWatch Logs data sources.

You can query CloudWatch Logs using three supported query language options:

Logs Insights QL - The AWS native query language specifically designed for CloudWatch Logs. It uses a SQL-like syntax with commands like fields, filter, stats, and sort. It’s optimized for the CloudWatch log structure and offers built-in functions for parsing timestamps, extracting fields from JSON logs, and performing aggregations.
OpenSearch SQL - OpenSearch SQL is a query language that uses a SQL-like syntax for querying data in OpenSearch. It supports standard SQL queries and is designed for users familiar with SQL.
OpenSearch PPL - The OpenSearch query language is based on Elasticsearch’s query DSL (Domain Specific Language). It uses a pipe-based syntax similar to Unix command-line tools or the Splunk search language, and supports complex boolean logic, range queries, wildcard matching, and full-text search capabilities.

Create a CloudWatch Logs query:

Select a region.
Select CloudWatch Logs from the query type drop-down.
Use the Logs Mode selector to choose between Logs Insights and Log Anomalies queries.

Log Anomalies

Anomaly detection uses machine-learning and pattern recognition to establish baselines of typical log content.
The Log Anomalies query editor fetches the list of anomalies detected in your CloudWatch service. In order to query log anomalies in the editor, a log anomaly detector must be created in the AWS CloudWatch console first. The log trend cell shows the number of occurrences of the pattern over the selected query time range. The table shows 50 log anomalies at a time. If you would like to narrow down the list, you can filter anomalies by their ARN and suppressed state.

In addition to this, you can use the Logs Insights QL editor and the anomaly command together with the patterns command to define and display log anomalies in real time. See the CloudWatch Logs Insights documentation for more info.

Logs Insights

The Query language drop-down is available only when Logs Mode is set to Logs Insights.

Select the query language you want to use in the Query language drop-down.
Specify the log groups, CloudWatch Logs data sources, or both that you want to query.
Use the main input area to write your logs query. Amazon CloudWatch only supports a subset of OpenSearch SQL and PPL commands. To find out more about the syntax supported, consult Amazon CloudWatch Logs documentation

Specify log groups to query

To target log groups in your query, use one of the following methods:

Log group name: Select specific log groups. This is the default behavior. You can select up to 50 log groups. Click Select log groups to choose the target log groups for your query. When the Monitoring account badge appears in the query editor header, you can search and select log groups across multiple accounts. Use the Accounts filter to narrow results by account. For large lists, use prefix search to narrow the selection. The selector loads log groups in pages, so you can scroll to load more results when many log groups are available.
Name prefix: Select the Name prefix query scope and provide up to five log group name prefixes. The query will run against log groups that have names that start with the specified prefixes. Each prefix must be at least three characters and must not include *. This option is only available for Logs Insights QL.
All log groups: Select the All log groups query scope. This queries all log groups in the selected region. This option is only available for Logs Insights QL.

You can also apply optional filters when specifying log groups with the Name prefix or All log groups query scopes:

Class: Filter by log group class. Choose Standard or Infrequent Access.
Accounts: Filter by AWS account. This option is only available when the Monitoring account badge appears in the query editor header. You can select up to 20 accounts.

Specify Amazon CloudWatch Logs data sources to query

Amazon CloudWatch Logs data sources categorize logs by the service or application that generates them and by the type of log. They complement log groups and let you query logs without knowing individual log group names. For details about how Amazon CloudWatch categorizes logs, refer to Data source discovery and management in the AWS documentation.

Before you use the data source selector, ensure that the IAM role or IAM user for the Grafana data source has the logs:ListAggregateLogGroupSummaries permission. For the complete IAM policy examples, refer to IAM policy examples.

To query logs by CloudWatch Logs data source, follow these steps:

Click Select data sources.
Search for a data source name or type.
Select up to 10 data sources.
Click Apply selection.

You can combine CloudWatch Logs data source selections with specific log groups or with the Name prefix and All log groups query scopes. When the Monitoring account badge appears, the selector includes data sources from the monitoring account and its linked source accounts.

The CloudWatch data source automatically adds the selected CloudWatch Logs data sources to Logs Insights QL and OpenSearch PPL queries.

Note
When you use Logs Insights QL, OpenSearch PPL, or OpenSearch SQL, you must specify a region and define the logs to query by selecting log groups, selecting CloudWatch Logs data sources, or choosing a supported log group query scope.

In OpenSearch SQL, you can specify log groups in multiple ways. For details, refer to the Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL section.

Click View in CloudWatch console to interactively view, search, and analyze your log data in the CloudWatch Logs Insights console. If you’re not logged in to the CloudWatch console, the link forwards you to the login page.

Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL

When querying log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL, you can use the $__source macro to automatically reference the log groups and CloudWatch Logs data sources selected in the query editor. This is the recommended approach as it allows you to manage both types of selection through the UI.

SELECT window.start, COUNT(*) AS exceptionCount
FROM `$__source`
WHERE `@message` LIKE '%Exception%'

The $__source macro expands to the proper syntax for the log groups and CloudWatch Logs data sources you’ve selected in the UI.

Alternatively, you can manually specify a single log group directly in the FROM clause:

SELECT window.start, COUNT(*) AS exceptionCount
FROM `log_group`
WHERE `@message` LIKE '%Exception%'

When querying multiple log groups you must use the logGroups(logGroupIdentifier: [...]) syntax:

SELECT window.start, COUNT(*) AS exceptionCount
FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`
WHERE `@message` LIKE '%Exception%'

To reference log groups in a monitoring account, use ARNs instead of LogGroup names.

You can also write queries returning time series data by using the stats command. When making stats queries in Explore, ensure you are in Metrics Explore mode.

Create queries for alerting

Alerting requires queries that return numeric data, which CloudWatch Logs supports. For example, you can enable alerts through the use of the stats command.

The following is a valid query for alerting on messages that include the text “Exception”:

filter @message like /Exception/
    | stats count(*) as exceptionCount by bin(1h)
    | sort exceptionCount desc

Note
If you receive an error like input data must be a wide series but got ... when trying to alert on a query, make sure that your query returns valid numeric data that can be output to a Time series panel.

For more information on Grafana alerts, refer to Alerting.

Common use cases

The following scenarios show how to combine the query editor’s features to solve real monitoring problems.

Monitor an EC2 fleet: Use a Metric Search query for AWS/EC2 CPUUtilization with the InstanceId dimension set to a wildcard (*) and Match Exact disabled. The graph automatically includes new instances as your fleet scales.
Track Lambda errors and invocations: Query the AWS/Lambda namespace for the Errors and Invocations metrics, then add a metric math expression such as errors / invocations * 100 to chart an error rate.
Rank the busiest resources: Use a Metrics Insights query with ORDER BY and LIMIT to return the top N resources by a metric, for example, the 10 EC2 instances with the highest average CPU utilization.
Investigate application errors: Use a CloudWatch Logs query with filter and stats to count errors over time, then correlate the spikes with metric panels on the same dashboard.
Alert on log error rates: Build a Logs query that returns numeric data with the stats command, then create an alert rule on the result. Refer to Create queries for alerting.

Cross-account observability

The CloudWatch plugin monitors and troubleshoots applications that span multiple accounts within a region. Cross-account observability enables seamless searching, visualization, and analysis of metrics and logs across account boundaries.

To enable cross-account observability, complete the following steps:

Go to the Amazon CloudWatch documentation and follow the instructions for enabling cross-account observability.
Add two API actions to the IAM policy attached to the role/user running the plugin.

Cross-account querying is available in the plugin through the Logs, Metric search, and Metric Insights modes. After you have configured it, you’ll see a Monitoring account badge in the query editor header.

Query caching

When you enable query and resource caching, Grafana temporarily stores the results of data source queries and resource requests. Query caching is available in CloudWatch Metrics in Grafana Cloud and Grafana Enterprise. It is not available in CloudWatch Logs Insights due to how query results are polled from AWS.

CloudWatch template variables

Tue, 18 Aug 2026 10:15:08 +0000

CloudWatch template variables

Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in drop-down select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.

For an introduction to templating and template variables, refer to Templating and Add and manage variables.

Use query variables

You can specify these CloudWatch data source queries in the Variable edit view’s Query type field. Use them to fill a variable’s options list with values like regions, namespaces, metric names, and dimension keys/values.

Name	List returned
Regions	All AWS regions.
Namespaces	All namespaces CloudWatch supports.
Metrics	Metrics in the namespace. (Specify region or use “default” for custom metrics.)
Dimension Keys	Dimension keys in the namespace.
Dimension Values	Dimension values matching the specified `region`, `namespace`, `metric`, and `dimension key`. Use dimension `filters` for more specific results.
EBS Volume IDs	Volume IDs matching the specified `region` and `instance ID`.
EC2 Instance Attributes	Attributes matching the specified `region`, `attribute name`, and `filters`.
Resource ARNs	ARNs matching the specified `region`, `resource type`, and `tags`.
Statistics	All standard statistics.
Log Groups	Log groups matching the specified `region` and optional log group `prefix`. The variable value is the log group ARN.
Accounts	The IDs of linked monitoring accounts for the specified `region`. Available only when cross-account observability is enabled.

When cross-account observability is enabled, the Metrics, Dimension Keys, Dimension Values, and Log Groups query types also show an optional Account field. Use it to target a specific linked monitoring account, or leave it unset to query all linked accounts. For more information, refer to Cross-account observability.

For details on the available dimensions, refer to the CloudWatch Metrics and Dimensions Reference.

For details about the metrics CloudWatch provides, refer to the CloudWatch documentation.

Use variables in queries

Use the Grafana variable syntax to include variables in queries. A query variable dynamically retrieves values from your data source using a query. For details, refer to the variable syntax documentation.

CloudWatch metric queries support variables in the Region, Namespace, Metric name, dimension key, and dimension value fields.

For example, if you create a variable named region with the Regions query type and a variable named instance with the EC2 Instance Attributes query type, you can reference them in a metric query by setting the query editor Region to $region and a dimension value to $instance. The dashboard then updates as you change the selected region and instance.

Query EC2 instance attributes

The EC2 Instance Attributes query type retrieves instance metadata using the AWS ec2:DescribeInstances API. Configure it with the following fields in the variable editor:

Region: The AWS region to query.
Attribute name: The instance attribute to return for each matching instance.
Filters: Optional. Filters that limit which instances are returned.

Note
The earlier ec2_instance_attribute(region, attributeName, filters) query syntax is still supported. Grafana automatically migrates existing variables that use it to the EC2 Instance Attributes query type.

Filters

Each filter has a key and a comma-separated list of matching values. Use the filters supported by ec2:DescribeInstances, such as instance properties. To filter by tag, use a key in the form tag:<name>, for example tag:Environment.

Select attributes

A query returns one attribute per instance. You can select any attribute that has a single value and isn’t an object or array, also known as a flat attribute. These attributes mirror the fields of the AWS EC2 Instance object, where you can find a definition of each one. Because the available attributes track the AWS API, the following list might not be exhaustive:

AmiLaunchIndex
Architecture
ClientToken
EbsOptimized
EnaSupport
Hypervisor
ImageId
InstanceId
InstanceLifecycle
InstanceType
KernelId
KeyName
LaunchTime
Platform
PrivateDnsName
PrivateIpAddress
PublicDnsName
PublicIpAddress
RamdiskId
RootDeviceName
RootDeviceType
SourceDestCheck
SpotInstanceRequestId
SriovNetSupport
SubnetId
VirtualizationType
VpcId

Some attributes are objects rather than flat values. To select a nested value, use dot notation, for example IamInstanceProfile.Arn.

To select a tag value, prepend the tag name with Tags.. For example, select the Name tag by using Tags.Name.

Amazon CloudWatch annotations

Tue, 18 Aug 2026 10:15:08 +0000

Amazon CloudWatch annotations

Annotations overlay rich event information on top of graphs. With the Amazon CloudWatch data source, annotations are based on CloudWatch alarm history. Each alarm history item within the dashboard time range—such as a state change, configuration update, or action—becomes an annotation, so you can correlate alarm activity with the rest of your dashboard data.

Before you begin

Before you create CloudWatch annotations, ensure you have:

A configured Amazon CloudWatch data source.
One or more CloudWatch alarms in the AWS account and region you want to query.
IAM permissions to describe alarms and alarm history. Refer to Required permissions.

How CloudWatch annotations work

Unlike data sources that build annotations from free-form queries, CloudWatch annotations query existing CloudWatch alarms and return their state-change history. Grafana finds the matching alarms using one of two methods, retrieves each alarm’s history with cloudwatch:DescribeAlarmHistory, and renders every history item as an annotation:

Match alarms by metric: Grafana calls cloudwatch:DescribeAlarmsForMetric to find alarms attached to a specific metric.
Match alarms by prefix: Grafana calls cloudwatch:DescribeAlarms with your alarm name and action prefixes, so AWS returns only alarms whose names and actions start with those prefixes. Grafana then filters the returned alarms by the namespace, metric name, dimensions, statistic, and period you specify.

Each annotation includes the alarm name as its title, the alarm history item type as a tag (for example, StateUpdate, ConfigurationUpdate, or Action), and the history summary as its text.

Create an annotation query

To add a CloudWatch annotation to a dashboard:

Open the dashboard where you want to add annotations.
Click Edit, then click Settings in the top navigation.
Select the Annotations tab.
Click Add annotation query.
Enter a Name for the annotation, for example, CloudWatch alarms.
Select your Amazon CloudWatch data source.
Configure the query fields described in the following sections.
Click Save dashboard.

The annotation query editor provides the following fields:

Field	Description
Region	The AWS region to query for alarms.
Namespace	The metric namespace, for example, `AWS/EC2`.
Metric name	The name of the metric the alarm watches, for example, `CPUUtilization`.
Statistic	The statistic the alarm uses, for example, `Average`.
Dimensions	The dimensions that identify the resource, for example, `InstanceId`.
Period	Optional. The minimum interval between data points, in seconds. Defaults to `300` when prefix matching is disabled.
Enable Prefix Matching	Optional. Match alarms by name and action prefix instead of by metric.
Action	Match only alarms whose actions start with this prefix. Available when prefix matching is enabled, and required for the query to run.
Alarm Name	Match only alarms whose names start with this prefix. Available when prefix matching is enabled, and required for the query to run.

Match alarms by metric

This is the default behavior, used when Enable Prefix Matching is off. Grafana returns annotations for any alarm attached to the metric you specify.

When you match by metric, Region, Namespace, Metric name, and Statistic are all required. If any of them are missing, the query returns an invalid annotations query error. Dimensions are optional, but adding them narrows the results to alarms on a specific resource.

For example, to annotate the history of the CPU utilization alarm on a single EC2 instance, set:

Region: us-east-1
Namespace: AWS/EC2
Metric name: CPUUtilization
Statistic: Average
Dimensions: InstanceId = i-0123456789abcdef0

Match alarms by prefix

Enable Enable Prefix Matching to find alarms by name and action prefix rather than by a specific metric. Grafana requests up to 100 alarms from cloudwatch:DescribeAlarms using the Alarm Name and Action prefixes, then filters the returned alarms by the namespace, metric name, dimensions, statistic, and period you specify. You must provide both an Alarm Name prefix and an Action prefix for the query to run.

Use prefix matching when you want to surface a group of related alarms, for example, all production alarms that trigger a specific SNS action. To do this, set Enable Prefix Matching on, then set Alarm Name to prod- and Action to the ARN prefix of your SNS topic, such as arn:aws:sns:us-east-1:123456789012:prod-.

Required permissions

Annotation queries require the following CloudWatch API actions in the IAM policy attached to the role or user running the data source:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingAlarmsFromCloudWatch",
      "Effect": "Allow",
      "Action": ["cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmsForMetric", "cloudwatch:DescribeAlarmHistory"],
      "Resource": "*"
    }
  ]
}

The metrics IAM policy on the configure page already includes these actions.

Troubleshoot annotations

If annotations don’t appear as expected, try the following solutions.

Annotations don’t appear

Verify the alarm has state changes within the dashboard time range. Annotations come from alarm history, so an alarm that never changed state produces no annotations.
Confirm the Region, Namespace, Metric name, Statistic, and Dimensions exactly match an existing alarm.
Check that the IAM role or user has the cloudwatch:DescribeAlarms, cloudwatch:DescribeAlarmsForMetric, and cloudwatch:DescribeAlarmHistory permissions.

`invalid annotations query` error

This error appears when prefix matching is disabled and the query reaches the backend with an empty Region, Namespace, Metric name, or Statistic. Provide all four fields, or enable Enable Prefix Matching.
In the dashboard annotation editor, an incomplete metric query is often skipped silently and produces no annotations rather than returning this error. If you see no annotations, confirm that all required fields are set.

Amazon CloudWatch alerting

Tue, 18 Aug 2026 10:15:08 +0000

Amazon CloudWatch alerting

The Amazon CloudWatch data source supports Grafana Alerting. You can create alert rules on CloudWatch metrics and CloudWatch Logs, so Grafana notifies you when a metric crosses a threshold or when a log query returns a value that meets your alert condition.

This is Grafana Alerting evaluated by Grafana, not CloudWatch alarms. To visualize the history of existing CloudWatch alarms instead, refer to Amazon CloudWatch annotations.

Before you begin

Before you create alert rules with CloudWatch data, ensure you have:

A configured Amazon CloudWatch data source.
Permissions to create alert rules in Grafana.
Familiarity with Grafana Alerting concepts.

Supported query types for alerting

Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. CloudWatch supports alerting for the following query types:

Query type	Use case	Notes
Metrics	Threshold-based alerts on CloudWatch metrics	Returns time series data; best suited for alerting.
Logs	Alert on log patterns, error counts, or thresholds	Use the `stats` command to aggregate logs into numeric values.

Note
You can’t create an alert rule from a metric math expression that references another query, such as queryA * 2. Define the metric directly in the alert query instead.

Create an alert rule

To create an alert rule using CloudWatch data:

Go to Alerting > Alert rules.
Click New alert rule.
Enter a name for your alert rule.
In the Define query and alert condition section:
- Select your Amazon CloudWatch data source.
- Configure your query, for example, a Metrics query for CPUUtilization or a Logs query that uses the stats command.
- Add a Reduce expression if your query returns multiple series.
- Add a Threshold expression to define the alert condition.
Configure the Set evaluation behavior section:
- Select or create a folder and evaluation group.
- Set the evaluation interval.
- Set the pending period.
Add labels and annotations to provide context for notifications.
Click Save rule.

For detailed instructions, refer to Create a Grafana-managed alert rule.

Example: metric threshold alert

This example fires when average EC2 CPU utilization exceeds 80%:

Create a new alert rule.
Configure the query:
- Namespace: AWS/EC2
- Metric name: CPUUtilization
- Statistic: Average
- Dimensions: InstanceId
Add expressions:
- Reduce: Last, to get the most recent data point.
- Threshold: Is above 80.
Set evaluation to run every 1 minute with a 5-minute pending period.
Save the rule.

Example: log error count alert

This example alerts when the number of log entries that contain Exception exceeds a threshold. CloudWatch Logs alerting requires a query that returns numeric data, which you can produce with the stats command:

Create a new alert rule.
Configure the query:
- Select CloudWatch Logs as the query mode.
- Select the region and log groups to query.
- Enter the following query:
```
filter @message like /Exception/
    | stats count(*) as exceptionCount by bin(5m)
```
Add expressions:
- Reduce: Max, to get the highest count in the period.
- Threshold: Is above 10.
Set evaluation to run every 5 minutes.
Save the rule.

Note
If you receive an error such as input data must be a wide series but got ..., make sure your query returns numeric data that can render in a time series panel.

CloudWatch Logs timeouts during alert evaluation

For CloudWatch Logs queries, Grafana polls AWS until the query completes or a timeout is reached. During alert evaluation, the timeout defined in the Grafana configuration file takes precedence over the Query Result Timeout setting on the data source. Set the configuration timeout high enough for your log queries to finish.

Best practices

Follow these recommendations to create reliable alerts with CloudWatch data.

Test queries before alerting

Verify your query returns numeric data before you create an alert:

Go to Explore.
Select your Amazon CloudWatch data source.
Run the query you plan to use for alerting.
Confirm the result is numeric and suitable for threshold evaluation.

Reduce multiple series

When a query returns multiple time series, use the Reduce expression to aggregate them into a single value with Last, Mean, Max, Min, or Sum.

Handle no data conditions

Configure how the rule behaves when no data is returned under Configure no data and error handling. Choose No Data, Alerting, or OK based on whether missing data should be treated as a problem.

Troubleshoot alerting

If your CloudWatch alerts don’t work as expected, use the following sections to diagnose common issues.

Alerts don’t fire

Confirm the query returns numeric data in Explore.
Ensure the evaluation interval allows enough time for data to be available.
Review the alert rule’s health and any error messages in the Alerting UI.

Log query alerts time out

Increase the Grafana configuration file timeout, which takes precedence during alert evaluation.
Narrow the time range or add filters to reduce the volume of data the query scans.

For more help, refer to Troubleshoot Amazon CloudWatch.

Troubleshoot Amazon CloudWatch data source issues

Tue, 18 Aug 2026 10:15:08 +0000

Troubleshoot Amazon CloudWatch data source issues

This document provides solutions to common issues you may encounter when configuring or using the Amazon CloudWatch data source. For configuration instructions, refer to Configure CloudWatch.

Note
The data source health check validates both metrics and logs permissions. If your IAM policy only grants access to one of these (for example, metrics-only or logs-only), the health check displays a red status. However, the service you have permissions for is still usable—you can query metrics or logs based on whichever permissions are configured.

Authentication errors

These errors occur when AWS credentials are invalid, missing, or don’t have the required permissions. Use the following table to match the error message you see to its most likely cause and the section that resolves it.

Error message	Likely cause	Refer to
`AccessDenied: User is not authorized to perform: cloudwatch:...`	The IAM policy is missing required CloudWatch permissions.	Access Denied or Not authorized to perform this operation
`InvalidClientTokenId` or `UnrecognizedClientException: The security token included in the request is invalid`	Credentials are invalid, deactivated, or rotated, or a transient Grafana Cloud incident is in progress.	InvalidClientTokenId or UnrecognizedClientException
`AccessDenied: ... not authorized to perform: sts:AssumeRole`	The role trust relationship, role ARN, or external ID is incorrect.	Unable to assume role
`AccessDenied: ... no VPC endpoint policy allows the cloudwatch:ListMetrics action`	A VPC endpoint policy or service control policy (SCP) blocks the action.	Access blocked by a VPC endpoint policy or service control policy

“Access Denied” or “Not authorized to perform this operation”

Symptoms:

Save & test fails with “Access Denied”
Queries return authorization errors
Namespaces, metrics, or dimensions don’t load

Possible causes and solutions:

Cause	Solution
IAM policy missing required permissions	Attach the appropriate IAM policy to your user or role. For metrics, you need `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, and related permissions. For logs, you need `logs:DescribeLogGroups`, `logs:StartQuery`, `logs:GetQueryResults`, and related permissions. Refer to Configure CloudWatch for complete policy examples.
Incorrect access key or secret key	Verify the credentials in the AWS Console under IAM > Users > your user > Security credentials. Generate new credentials if necessary.
Credentials have expired	For temporary credentials, generate new ones. For access keys, verify they haven’t been deactivated or deleted.
Wrong AWS region	Verify the default region in the data source configuration matches where your resources are located.
IAM role lacks permissions in the queried region	AWS evaluates permissions per region. If your role only has CloudWatch access in some regions, querying another region returns `AccessDenied`, and the error might not name the region. Set the data source Default Region to a region where your role has permissions, and only query regions your IAM policy allows.
Assume Role ARN is incorrect	Verify the role ARN format: `arn:aws:iam::<account-id>:role/<role-name>`. Check that the role exists in the AWS Console.

“InvalidClientTokenId” or “UnrecognizedClientException”

Symptoms:

Save & test fails with InvalidClientTokenId: The security token included in the request is invalid
Queries fail with UnrecognizedClientException: The security token included in the request is invalid
The error appears suddenly without any configuration change

Possible causes and solutions:

Cause	Solution
Access key or secret key is invalid or mistyped	Verify the credentials in the AWS Console under IAM > Users > your user > Security credentials. Re-enter them in the data source configuration.
Credentials were deactivated, deleted, or rotated	Generate new credentials in AWS and update the data source configuration.
Temporary credentials expired	Generate new temporary credentials, or switch to a long-lived authentication method such as Assume Role.
Transient Grafana Cloud incident	If multiple data sources fail at the same time with `InvalidClientTokenId` and your credentials are unchanged, check the Grafana Cloud status page for an ongoing incident before you rotate credentials.

“Unable to assume role”

Symptoms:

Authentication fails when using Assume Role ARN
Error references STS or AssumeRole, for example, AccessDenied: User is not authorized to perform: sts:AssumeRole

Solutions:

Verify the trust relationship on the IAM role allows the Grafana credentials to assume it.
Check the trust policy includes the correct principal (the user or role running Grafana).
If you use the Grafana Assume Role authentication method, confirm the trust policy uses the exact Grafana AWS account ID and external ID shown in the instructions box on the data source’s Settings tab. Entering a different account ID is a common mistake that causes the assume-role request to fail.
If using an external ID, ensure it matches exactly in both the role’s trust policy and the Grafana data source configuration.
Verify the base credentials have the sts:AssumeRole permission.
Check that the role ARN is correct and the role exists.

Example trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<your-account-id>:user/<grafana-user>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<your-external-id>"
        }
      }
    }
  ]
}

“Save & test” passes but queries return no data

Symptoms:

Save & test succeeds, but panels and the query editor return no data
Namespaces, metrics, or dimensions don’t load even though the connection test passed
The issue appears only when you use an Assume Role ARN

Possible causes and solutions:

Cause	Solution
The connection test only validates the assume-role step	When you configure an Assume Role ARN, Save & test confirms that the primary credentials can perform `sts:AssumeRole`. It doesn’t verify that the assumed role has CloudWatch or Logs permissions. Attach the required query permissions to the assumed role, not only to the primary credentials.
The assumed role is missing CloudWatch permissions	Add the metrics and logs actions, such as `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, `logs:DescribeLogGroups`, and `logs:StartQuery`, to the assumed role’s policy. Refer to Configure CloudWatch for complete policy examples.
Permissions are attached to the wrong identity	Confirm that the policy is attached to the role named in Assume Role ARN, not to the IAM user or role that provides the primary credentials.

Data source stops working after about an hour

Symptoms:

The data source works initially, then queries fail after roughly an hour
Errors reference an expired or invalid security token
Saving the data source again, or restarting Grafana, temporarily restores access until it fails again

Possible causes and solutions:

Cause	Solution
Temporary credentials expired and weren’t refreshed	Methods that use temporary credentials, such as Assume Role or STS, issue short-lived tokens. If the data source uses static temporary credentials, switch to an authentication method that refreshes automatically, such as Assume Role with long-lived base credentials or an EKS/EC2 instance role. Refer to AWS authentication.

Access blocked by a VPC endpoint policy or service control policy

Symptoms:

Credentials are valid and the IAM policy grants the action, but requests still fail with AccessDenied
The error references a VPC endpoint policy, for example, AccessDenied: ... no VPC endpoint policy allows the cloudwatch:ListMetrics action
Access fails only from within a VPC or in a restricted account, such as a FedRAMP environment

Possible causes and solutions:

Cause	Solution
A VPC endpoint policy restricts which actions are allowed	Update the policy on the CloudWatch, Logs, EC2, or STS VPC endpoints to allow the required actions, such as `cloudwatch:ListMetrics` and `cloudwatch:GetMetricData`.
A service control policy (SCP) explicitly denies a required action	Review the service control policies applied to the account or organizational unit. An explicit `Deny` in an SCP overrides any `Allow` in an IAM policy, so the action must be permitted at both levels.
Requests route through the wrong endpoint	Confirm that the data source region and any custom endpoint match the VPC endpoints that permit the required actions.

AWS SDK Default authentication not working

Symptoms:

Data source test fails when using AWS SDK Default
Works locally but fails in production

Solutions:

Verify AWS credentials are configured in the environment where Grafana runs.
Check for credentials in the default locations:
- Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
- Shared credentials file (~/.aws/credentials)
- EC2 instance metadata (if running on EC2)
- ECS task role (if running in ECS)
- EKS service account (if running in EKS)
Ensure the Grafana process has permission to read the credentials file.
For EKS with IRSA, set the pod’s security context to allow user 472 (grafana) to access the projected token. Refer to AWS authentication for details.

Credentials file not found

Symptoms:

Error indicates credentials file cannot be read
Authentication fails with “Credentials file” option

Solutions:

Create the credentials file at ~/.aws/credentials for the user running the grafana-server service.
Verify the file has correct permissions (0644).
If the file exists but isn’t working, move it to /usr/share/grafana/ and set permissions to 0644.
Ensure the profile name in the data source configuration matches a profile in the credentials file.

Connection errors

These errors occur when Grafana cannot reach AWS CloudWatch endpoints.

“Request timed out” or connection failures

Symptoms:

Data source test times out
Queries fail with timeout errors
Intermittent connection issues

Solutions:

Verify network connectivity from the Grafana server to AWS endpoints.
Check firewall rules allow outbound HTTPS (port 443) to AWS services.
If using a VPC, ensure proper NAT gateway or VPC endpoint configuration.
For Grafana Cloud connecting to private resources, configure Private data source connect.
Check if the default region is correct—incorrect regions may cause longer timeouts.
Increase the timeout settings if queries involve large data volumes.

Custom endpoint configuration issues

Symptoms:

Connection fails when using a custom endpoint
Endpoint URL rejected

Solutions:

Verify the endpoint URL format is correct.
Ensure the endpoint is accessible from the Grafana server.
Check that the endpoint supports the required AWS APIs.
For VPC endpoints, verify the endpoint policy allows the required actions.

CloudWatch Metrics query errors

These errors occur when querying CloudWatch Metrics.

“No data” or empty results

Symptoms:

Query executes without error but returns no data
Charts show “No data” message

Possible causes and solutions:

Cause	Solution
Time range doesn’t contain data	Expand the dashboard time range. CloudWatch metrics have different retention periods based on resolution.
Wrong namespace or metric name	Verify the namespace (for example, `AWS/EC2`) and metric name (for example, `CPUUtilization`) are correct.
Incorrect dimensions	Ensure dimension names and values match your AWS resources exactly.
Match Exact enabled incorrectly	When Match Exact is enabled, all dimensions must be specified. Try disabling it to see if metrics appear.
Period too large	Reduce the period setting or set it to “auto” to ensure data points are returned for your time range.
Custom metrics not configured	Add custom metric namespaces in the data source configuration under Namespaces of Custom Metrics.

“Metric not found” or metrics don’t appear in drop-down

Symptoms:

Expected metrics don’t appear in the query editor
Metric drop-down is empty for a namespace
A specific metric is missing from the drop-down even though it exists in the CloudWatch console

Solutions:

Verify the metric exists in the selected region.
For custom metrics, add the namespace to Namespaces of Custom Metrics in the data source configuration.
Check that the IAM policy includes cloudwatch:ListMetrics permission.
CloudWatch limits ListMetrics to 500 results per page. To retrieve more metrics, increase the list_metrics_page_limit setting in the Grafana configuration file.
Type the metric name directly into the Metric name field. For some AWS namespaces, the data source uses a predefined metric list, so newer or less common metrics might not appear in the drop-down even though you can still query them. If the metric exists in your CloudWatch console, enter its exact name to query it.
Use the Query Inspector to verify the API request and response.

Dimension values not loading

Symptoms:

Dimension value drop-down doesn’t populate
Wildcard searches return no results

Solutions:

Verify the IAM policy includes cloudwatch:ListMetrics permission.
Check that the namespace and metric are selected before dimension values can load.
For EC2 dimensions, ensure ec2:DescribeTags and ec2:DescribeInstances permissions are granted.
Dimension values require existing metrics—if no metrics match, no values appear.

“Too many data points” or API throttling

Symptoms:

Queries fail with throttling errors
Performance degrades with multiple panels

Solutions:

Increase the period setting to reduce the number of data points.
Reduce the time range of your queries.
Use fewer dimensions or wildcard queries per panel.
Request a quota increase for GetMetricData requests per second in the AWS Service Quotas console.
Enable query caching in Grafana to reduce API calls.

This section covers throttling at the query level. For account-wide rate limits and quota increases, refer to API throttling errors.

Metric math expression errors

Symptoms:

Expression returns errors
Referenced metrics not found

Solutions:

Verify each referenced metric has a unique ID set.
Check that metric IDs start with a lowercase letter and contain only letters, numbers, and underscores.
Ensure all referenced metrics are in the same query.
Verify the expression syntax follows AWS Metric Math documentation.
Metric math expressions can’t be used with Grafana alerting if they reference other query rows.

CloudWatch Logs query errors

These errors occur when querying CloudWatch Logs.

“Query failed” or logs don’t appear

Symptoms:

Log queries return errors
No log data is displayed

Solutions:

Verify log group names are correct and exist in the selected region.
Check the IAM policy includes logs:StartQuery, logs:GetQueryResults, and logs:DescribeLogGroups permissions.
Ensure the time range contains log data.
Verify the query syntax is valid. For CloudWatch Logs Insights QL, test the query in the AWS Console.
Select the correct query language (Logs Insights QL, OpenSearch PPL, or OpenSearch SQL) based on your query syntax.

Log query timeout

Symptoms:

Query runs for a long time then fails
Error mentions timeout

Solutions:

Increase the Query Result Timeout setting in the data source configuration (default is 30 minutes).
Narrow the time range to reduce the amount of data scanned.
Add filters to your query to limit results.
Break complex queries into smaller, more focused queries.
For alerting, the timeout defined in the Grafana configuration file takes precedence.

Log groups not appearing in selector

Symptoms:

Log group selector is empty
Can’t find expected log groups

Solutions:

Verify the IAM policy includes logs:DescribeLogGroups permission.
Check that log groups exist in the selected region.
For cross-account observability, ensure proper IAM permissions for oam:ListSinks and oam:ListAttachedLinks.
Use prefix search to filter log groups if you have many groups.
Verify the selected account (for cross-account) contains the expected log groups.

OpenSearch SQL query errors

Symptoms:

OpenSearch SQL queries fail
Syntax errors with SQL queries

Solutions:

Specify the log group identifier or ARN in the FROM clause:
SQL
```
SELECT * FROM `log_group_name` WHERE `@message` LIKE '%error%'
```

For multiple log groups, use the logGroups function:

SELECT * FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`

Amazon CloudWatch supports only a subset of OpenSearch SQL commands. Refer to the CloudWatch Logs documentation for supported syntax.

Template variable errors

These errors occur when using template variables with the CloudWatch data source.

Variables return no values

Symptoms:

Variable drop-down is empty
Dashboard fails to load with variable errors

Solutions:

Verify the data source connection is working.
Check that the IAM policy includes permissions for the variable query type:
- Regions: No additional permissions needed.
- Namespaces: No additional permissions needed.
- Metrics: Requires cloudwatch:ListMetrics.
- Dimension Values: Requires cloudwatch:ListMetrics.
- EC2 Instance Attributes: Requires ec2:DescribeInstances.
- EBS Volume IDs: Requires ec2:DescribeVolumes.
- Resource ARNs: Requires tag:GetResources.
- Log Groups: Requires logs:DescribeLogGroups.
For dependent variables, ensure parent variables have valid selections.
Verify the region is set correctly (use “default” for the data source’s default region).

For more information on template variables, refer to CloudWatch template variables.

Multi-value template variables cause query failures

Symptoms:

Queries fail when selecting multiple dimension values
Error about search expression limits

Solutions:

Search expressions are limited to 1,024 characters. Reduce the number of selected values.
Use the asterisk (*) wildcard instead of selecting “All” to query all metrics for a dimension.
Multi-valued template variables are only supported for dimension values—not for Region, Namespace, or Metric Name.

Cross-account observability errors

These errors occur when using CloudWatch cross-account observability features.

Cross-account queries fail

Symptoms:

Can’t query metrics or logs from linked accounts
Monitoring account badge doesn’t appear

Solutions:

Verify cross-account observability is configured in the AWS CloudWatch console.

Add the required IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Check that the monitoring account and source accounts are properly linked in AWS.
Cross-account observability works within a single region—verify all accounts are in the same region.
EC2 Instance Attributes can’t be queried across accounts because they use the EC2 API, not the CloudWatch API.

Quota and pricing issues

These issues relate to AWS service quotas and cost management.

API throttling errors

Symptoms:

“Rate exceeded” errors
Dashboard panels intermittently fail to load

Solutions:

Reduce the frequency of dashboard refreshes.
Increase the period setting to reduce GetMetricData requests.
Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud).
Request a quota increase in the AWS Service Quotas console.
Consider consolidating similar queries using metric math.

If throttling happens because a single query returns too many data points, refer to “Too many data points” or API throttling.

Unexpectedly high CloudWatch costs

Symptoms:

AWS CloudWatch costs are higher than expected
Frequent API calls from Grafana

Solutions:

The GetMetricData API doesn’t qualify for the CloudWatch API free tier.
Reduce dashboard auto-refresh frequency.
Increase the period setting to reduce data points returned.
Use query caching to reduce repeated API calls.
Review variable query settings—set variable refresh to “On dashboard load” instead of “On time range change.”
Avoid using wildcards in dimensions when possible, as they generate search expressions with multiple API calls.

Other common issues

These issues don’t produce specific error messages but are commonly encountered.

Custom metrics don’t appear

Symptoms:

Custom metrics from applications or agents don’t show in the namespace drop-down
Only standard AWS namespaces are visible

Solutions:

Add your custom metric namespace to the Namespaces of Custom Metrics field in the data source configuration.
Separate multiple namespaces with commas (for example, CWAgent,CustomNamespace).
Verify custom metrics have been published to CloudWatch in the selected region.

Pre-configured dashboards not working

Symptoms:

Imported dashboards show no data
Dashboard variables don’t load

Solutions:

Verify the data source name in the dashboard matches your CloudWatch data source.
Check that the dashboard’s AWS region setting matches where your resources are located.
Ensure the IAM policy grants access to the required services (EC2, Lambda, RDS, etc.).
Verify resources exist and are emitting metrics in the selected region.

Application Signals trace links not appearing

Symptoms:

Log entries don’t show Application Signals trace links
@xrayTraceId field not appearing

Solutions:

Verify an Application Signals data source is configured and linked in the CloudWatch data source settings.
Ensure your logs contain the @xrayTraceId field.
Update log queries to include @xrayTraceId in the fields, for example: fields @message, @xrayTraceId.
Configure your application to log Application Signals trace IDs. Refer to the AWS Application Signals documentation.

Enable debug logging

To capture detailed error information for troubleshooting:

Set the Grafana log level to debug in the configuration file:
ini
```
[log]
level = debug
```
Review logs in /var/log/grafana/grafana.log (or your configured log location).
Look for CloudWatch-specific entries that include request and response details.
Reset the log level to info after troubleshooting to avoid excessive log volume.

Get additional help

If you’ve tried the solutions above and still encounter issues:

Check the Grafana community forums for similar issues.
Review the CloudWatch plugin GitHub issues for known bugs.
Consult the AWS CloudWatch documentation for service-specific guidance.
Contact Grafana Support if you’re an Enterprise, Cloud Pro, or Cloud Contracted user.
When reporting issues, include:
- Grafana version
- AWS region
- Error messages (redact sensitive information)
- Steps to reproduce
- Query configuration (redact credentials and account IDs)

Amazon CloudWatch data source on Grafana Labs

Configure CloudWatch

Configure the Amazon CloudWatch data source

Before you begin

Key concepts

Add the CloudWatch data source

Configure the data source in the UI

IAM policy examples

Cross-account observability permissions

Permissions reference

CloudWatch Logs data protection

Configure the data source with grafana.ini

Provision the data source

Configure with Terraform

Configure AWS authentication

Configure AWS authentication

Select an authentication method

Assume a role

Use an external ID

Use a custom endpoint

Use an AWS credentials file

Use EKS IAM roles for service accounts

Use Grafana Assume Role

Amazon CloudWatch query editor

Amazon CloudWatch query editor

Key concepts

Choose a query editing mode

CloudWatch Metrics query editor components

Use Builder and Code modes

Builder mode

Code mode

Query CloudWatch metrics

Metric Search queries

Create dynamic queries with dimension wildcards

Use multi-value template variables

Use metric math expressions

Query metrics across AWS monitoring accounts

Period macro

Deep-link Grafana panels to the CloudWatch console

Use Metric Insights syntax

Use Metrics Insights keywords

Query CloudWatch Logs

Specify log groups to query

Specify Amazon CloudWatch Logs data sources to query

Query log groups and Amazon CloudWatch Logs data sources with OpenSearch SQL

Create queries for alerting

Common use cases

Cross-account observability

Query caching

CloudWatch template variables

CloudWatch template variables

Use query variables

Use variables in queries

Query EC2 instance attributes

Filters

Select attributes

Amazon CloudWatch annotations

Amazon CloudWatch annotations

Before you begin

How CloudWatch annotations work

Create an annotation query

Match alarms by metric

Match alarms by prefix

Required permissions

Troubleshoot annotations

Annotations don’t appear

invalid annotations query error

Related resources

Amazon CloudWatch alerting

Amazon CloudWatch alerting

Before you begin

Supported query types for alerting

Create an alert rule

Example: metric threshold alert

Example: log error count alert

CloudWatch Logs timeouts during alert evaluation

Best practices

Test queries before alerting

Reduce multiple series

Handle no data conditions

`invalid annotations query` error