CloudWatch logs integration for Grafana Cloud
The CloudWatch logs integration enables you to send CloudWatch logs to Grafana Cloud. The integration will guide you through the deployment of a Lambda function that forwards CloudWatch logs to Grafana Cloud Loki.
Install CloudWatch logs integration for Grafana Cloud
In your Grafana Cloud instance, click on Connections from the Home menu located near the top left-hand side of the page.
Navigate to the CloudWatch logs tile and click on it to review the prerequisites and follow the installation instructions.
Automatically configure lambda-promtail using CloudFormation
Follow the installation instructions to deploy lambda-promtail
in your AWS account.
Upload the zipfile with the
lambda-promtail
binary to an S3 bucket in the region where you want to pull CloudWatch logs from.Create a Grafana.com API key with the
MetricsPublisher
role to authenticate with Grafana Cloud Loki.Launch the CloudFormation stack. Enter the Grafana.com API key created in the previous step and specify the
SubscriptionFilter
for the Log Group you want to pull from.Once the Lambda is running, navigate to the Grafana Explore view to verify logs are being forwarded correctly. Incoming logs can be identified by the
__aws_cloudwatch_log_group
label.
Automatically configure lambda-promtail using Terraform
You’ll find a Terraform snippet in this section that can be used to provision all resources necessary to deploy lambda-promtail
in your AWS account.
To run the Terraform setup:
Configure the AWS CLI. Remember to set the correct AWS region where
lambda-promtail
should run and pull CloudWatch logs from.Copy this snippet into a Terraform file:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
}
}
data "aws_region" "current" {}
resource "aws_s3_object_copy" "lambda_promtail_zipfile" {
bucket = var.s3_bucket
key = var.s3_key
source = "grafanalabs-cf-templates/lambda-promtail/lambda-promtail.zip"
}
resource "aws_iam_role" "lambda_promtail_role" {
name = "GrafanaLabsCloudWatchLogsIntegration"
assume_role_policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Action" : "sts:AssumeRole",
"Principal" : {
"Service" : "lambda.amazonaws.com"
},
"Effect" : "Allow",
}
]
})
}
resource "aws_iam_role_policy" "lambda_promtail_policy_logs" {
name = "lambda-logs"
role = aws_iam_role.lambda_promtail_role.name
policy = jsonencode({
"Statement" : [
{
"Action" : [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
],
"Effect" : "Allow",
"Resource" : "arn:aws:logs:*:*:*",
}
]
})
}
resource "aws_cloudwatch_log_group" "lambda_promtail_log_group" {
name = "/aws/lambda/GrafanaCloudLambdaPromtail"
retention_in_days = 14
}
resource "aws_lambda_function" "lambda_promtail" {
function_name = "GrafanaCloudLambdaPromtail"
role = aws_iam_role.lambda_promtail_role.arn
timeout = 60
memory_size = 128
handler = "main"
runtime = "go1.x"
s3_bucket = var.s3_bucket
s3_key = var.s3_key
environment {
variables = {
WRITE_ADDRESS = var.write_address
USERNAME = var.username
PASSWORD = var.password
KEEP_STREAM = var.keep_stream
BATCH_SIZE = var.batch_size
EXTRA_LABELS = var.extra_labels
}
}
depends_on = [
aws_s3_object_copy.lambda_promtail_zipfile,
aws_iam_role_policy.lambda_promtail_policy_logs,
aws_cloudwatch_log_group.lambda_promtail_log_group,
]
}
resource "aws_lambda_function_event_invoke_config" "lambda_promtail_invoke_config" {
function_name = aws_lambda_function.lambda_promtail.function_name
maximum_retry_attempts = 2
}
resource "aws_lambda_permission" "lambda_promtail_allow_cloudwatch" {
statement_id = "lambda-promtail-allow-cloudwatch"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.lambda_promtail.function_name
principal = "logs.${data.aws_region.current.name}.amazonaws.com"
}
# This block allows for easily subscribing to multiple log groups via the `log_group_names` var.
# However, if you need to provide an actual filter_pattern for a specific log group you should
# copy this block and modify it accordingly.
resource "aws_cloudwatch_log_subscription_filter" "lambda_promtail_logfilter" {
for_each = toset(var.log_group_names)
name = "lambda_promtail_logfilter_${each.value}"
log_group_name = each.value
destination_arn = aws_lambda_function.lambda_promtail.arn
# required but can be empty string
filter_pattern = ""
depends_on = [aws_iam_role_policy.lambda_promtail_policy_logs]
}
output "role_arn" {
value = aws_lambda_function.lambda_promtail.arn
description = "The ARN of the Lambda function that runs lambda-promtail."
}
- Copy the following snippet into a
variables.tf
file. You’ll need to paste here some of the values displayed in the installation instructions (e.g. thewrite_address
,username
andpassword
).
variable "write_address" {
type = string
description = "This is the Grafana Cloud Loki URL that logs will be forwarded to."
default = ""
}
variable "username" {
type = string
description = "The basic auth username for Grafana Cloud Loki."
default = ""
}
variable "password" {
type = string
description = "The basic auth password for Grafana Cloud Loki (your Grafana.com API Key)."
sensitive = true
default = ""
}
variable "s3_bucket" {
type = string
description = "The name of the bucket where to upload the 'lambda-promtail.zip' file."
default = ""
}
variable "s3_key" {
type = string
description = "The desired path where to upload the 'lambda-promtail.zip' file (defaults to the root folder)."
default = "lambda-promtail.zip"
}
variable "log_group_names" {
type = list(string)
description = "List of CloudWatch Log Group names to create Subscription Filters for (ex. /aws/lambda/my-log-group)."
default = []
}
variable "keep_stream" {
type = string
description = "Determines whether to keep the CloudWatch Log Stream value as a Loki label when writing logs from lambda-promtail."
default = "false"
}
variable "extra_labels" {
type = string
description = "Comma separated list of extra labels, in the format 'name1,value1,name2,value2,...,nameN,valueN' to add to entries forwarded by lambda-promtail."
default = ""
}
variable "batch_size" {
type = string
description = "Determines when to flush the batch of logs (bytes)."
default = ""
}
- Configure variables according to their descriptions. Note that all resources must be in the same AWS region (CloudWatch Log Group, Lambda function, S3 bucket for
lambda-promtail.zip
). Finally, run theterraform apply
command:
terraform apply -var-file="variables.tf"
Once the terraform apply
command has finished creating the resources, it will output the role_arn
of the Lambda function that runs lambda-promtail
.
The Terraform snippets above should get you started with a basic configuration for lambda-promtail
. For additional setup (e.g. VPC subnets and security groups) read through this extended example Terraform file.
Log labels
CloudWatch logs forwarded to Grafana Cloud Loki the following special labels assigned to them:
__aws_cloudwatch_log_group
: The associated Cloudwatch Log Group for this log.__aws_cloudwatch_owner
: The AWS ID of the owner of this log.__aws_cloudwatch_log_stream
: The associated Cloudwatch Log Stream for this log (ifKEEP_STREAM
is set totrue
).
Both the CloudFormation and Terraform setup allow to specify “extra labels” (as key-value pairs) that will be added to logs streamed by lambda-promtail
. These extra labels will take the form __extra_<name>=<value>
.
Related resources from Grafana Labs


