Configure Amazon CloudWatch logs with Lambda

Send logs from Amazon CloudWatch Logs to AWS observability in Grafana Cloud Provider using a Lambda Promtail function and a subscription filter that invokes the Lambda function when new log events are written.

For supported log types and a consolidated label reference, refer to Logs with Lambda. For advanced options (Amazon Kinesis, relabeling, Amazon ECR images), refer to the Lambda Promtail client documentation.

Before you begin

Before configuring AWS logs with Lambda Promtail, make sure you have:

A Grafana Cloud account with a Grafana Cloud stack.
The Cloud Provider AWS Writer app plugin role or Grafana Cloud Admin basic role.
An AWS account with permissions to create Lambda functions, IAM roles, and either CloudWatch subscription filters or S3 notifications in the same AWS Region as your log source.
The AWS CLI configured.
The Grafana Cloud Configuration details values for Loki (Grafana Cloud Loki write address and username, and a Grafana Cloud access policy token), available on the Logs with Lambda page in Grafana Cloud Provider.

Navigate to Logs with Lambda

Before you can begin configuring AWS logs with Lambda Promtail, you must navigate to the Logs with Lambda page in Grafana Cloud Provider.

To navigate to the Logs with Lambda page:

Open your Grafana Cloud stack.
Expand Observability > Cloud provider > AWS in the main menu of your Grafana Cloud stack.
Click the Configuration tab.
Click the Logs with Lambda tile.

Then continue with CloudFormation or Terraform.

Configure with CloudFormation

The provided CloudFormation template can get you started quickly and requires little additional configuration. CloudFormation is a great option for AWS-native infrastructure definitions.

Upload Lambda Promtail

Before launching the CloudFormation stack, you need to upload the Lambda Promtail compressed binary file to the AWS S3 bucket in the same region as the AWS S3 bucket with your CloudWatch log groups.

To upload the Lambda Promtail compressed binary to your S3 bucket:

On the Logs with Lambda page in Grafana Cloud Provider, click Use CloudFormation.
Copy the upload command from the Grafana Cloud UI.
Update the command with YOUR-BUCKET-NAME and YOUR-REGION-NAME and run it in your terminal.
This command uploads the compressed lambda-promtail build to an S3 bucket in the same AWS Region where the Lambda function runs and where your CloudWatch log groups are stored.

Create a Grafana Cloud access policy token

Create a Grafana Cloud access policy token with the MetricsPublisher role so Lambda Promtail can authenticate to Grafana Cloud.

To generate a Grafana Cloud access policy token with MetricsPublisher role:

On the Logs with Lambda page in Grafana Cloud Provider, in the API token name field, enter a name for the token.
Select an Expiration date.
We recommend you select a specific expiration date and do not set the Expiration date to No expiry, as this can create a security vulnerability.
Click Create token.
Copy the generated key and store it securely.

Launch the CloudFormation stack

After you have the Grafana Cloud access policy token, you must launch and customize the CloudFormation stack template. The AWS account you are logged into is the account where resources are created. To use a different account, log out and log into the desired account before launching the CloudFormation stack.

To launch and customize the CloudFormation stack:

On the Logs with Lambda page in Grafana Cloud Provider, click Launch CloudFormation stack.
In the AWS CloudFormation console, paste the Grafana Cloud access policy token into the Password parameter (or the field your template uses for Loki basic auth).
Configure the LogGroup and SubscriptionFilter parameters so the filter targets the log groups you want to ship.

Configure with Terraform

The provided Terraform sample code snippets can get you started quickly and require little additional configuration.

To configure logs with Lambda using Terraform, on the Logs with Lambda page in Grafana Cloud Provider click Use Terraform.

Create a Grafana Cloud access policy token

Create a Grafana Cloud access policy token with the MetricsPublisher role so Lambda Promtail can authenticate to Grafana Cloud.

To generate a Grafana Cloud access policy token with MetricsPublisher role:

On the Logs with Lambda page in Grafana Cloud Provider, in the API token name field, enter a name for the token.
Select an Expiration date.
We recommend you select a specific expiration date and do not set the Expiration date to No expiry, as this can create a security vulnerability.
Click Create token.
Copy the generated key and store it securely.

Terraform setup

Configure the AWS CLI for the Region where Lambda, your log groups, and the S3 bucket for the zip artifact live.

Copy the following Terraform snippet into a main.tf file (or merge into your existing root module):

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

data "aws_region" "current" {}

resource "aws_s3_object_copy" "lambda_promtail_zipfile" {
  bucket = var.s3_bucket
  key    = var.s3_key
  source = "grafanalabs-cf-templates/lambda-promtail/lambda-promtail.zip"
}

resource "aws_iam_role" "lambda_promtail_role" {
  name = "GrafanaLabsCloudWatchLogsIntegration"

  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Action" : "sts:AssumeRole",
        "Principal" : {
          "Service" : "lambda.amazonaws.com"
        },
        "Effect" : "Allow"
      }
    ]
  })
}

resource "aws_iam_role_policy" "lambda_promtail_policy_logs" {
  name = "lambda-logs"
  role = aws_iam_role.lambda_promtail_role.name
  policy = jsonencode({
    "Statement" : [
      {
        "Action" : [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        "Effect" : "Allow",
        "Resource" : "arn:aws:logs:*:*:*"
      }
    ]
  })
}

resource "aws_cloudwatch_log_group" "lambda_promtail_log_group" {
  name              = "/aws/lambda/GrafanaCloudLambdaPromtail"
  retention_in_days = 14
}

resource "aws_lambda_function" "lambda_promtail" {
  function_name = "GrafanaCloudLambdaPromtail"
  role          = aws_iam_role.lambda_promtail_role.arn

  timeout     = 60
  memory_size = 128

  handler   = "main"
  runtime   = "provided.al2023"
  s3_bucket = var.s3_bucket
  s3_key    = var.s3_key

  environment {
    variables = {
      WRITE_ADDRESS = var.write_address
      USERNAME      = var.username
      PASSWORD      = var.password
      KEEP_STREAM   = var.keep_stream
      BATCH_SIZE    = var.batch_size
      EXTRA_LABELS  = var.extra_labels
    }
  }

  depends_on = [
    aws_s3_object_copy.lambda_promtail_zipfile,
    aws_iam_role_policy.lambda_promtail_policy_logs,
    aws_cloudwatch_log_group.lambda_promtail_log_group
  ]
}

resource "aws_lambda_function_event_invoke_config" "lambda_promtail_invoke_config" {
  function_name          = aws_lambda_function.lambda_promtail.function_name
  maximum_retry_attempts = 2
}

resource "aws_lambda_permission" "lambda_promtail_allow_cloudwatch" {
  statement_id  = "lambda-promtail-allow-cloudwatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.lambda_promtail.function_name
  principal     = "logs.${data.aws_region.current.name}.amazonaws.com"
}

# Subscribe each log group in var.log_group_names. For a different filter_pattern per group,
# duplicate this resource and set filter_pattern explicitly.
resource "aws_cloudwatch_log_subscription_filter" "lambda_promtail_logfilter" {
  for_each        = toset(var.log_group_names)
  name            = "lambda_promtail_logfilter_${each.value}"
  log_group_name  = each.value
  destination_arn = aws_lambda_function.lambda_promtail.arn
  filter_pattern  = ""
  depends_on      = [aws_iam_role_policy.lambda_promtail_policy_logs]
}

output "lambda_arn" {
  value       = aws_lambda_function.lambda_promtail.arn
  description = "ARN of the Lambda function that runs lambda-promtail."
}

Copy the following Terraform snippet into variables.tf:

variable "write_address" {
  type        = string
  description = "Grafana Cloud Loki push URL."
  default     = ""
}

variable "username" {
  type        = string
  description = "Basic auth username for Grafana Cloud Loki."
  default     = ""
}

variable "password" {
  type        = string
  description = "Basic auth password for Grafana Cloud Loki (Grafana.com API key)."
  sensitive   = true
  default     = ""
}

variable "s3_bucket" {
  type        = string
  description = "Bucket that holds lambda-promtail.zip."
  default     = ""
}

variable "s3_key" {
  type        = string
  description = "Object key for lambda-promtail.zip."
  default     = "lambda-promtail.zip"
}

variable "log_group_names" {
  type        = list(string)
  description = "CloudWatch log group names to subscribe (for example /aws/lambda/my-function)."
  default     = []
}

variable "keep_stream" {
  type        = string
  description = "If true, keep the CloudWatch log stream name as a Loki label (can increase cardinality)."
  default     = "false"
}

variable "extra_labels" {
  type        = string
  description = "Comma-separated pairs: name1,value1,name2,value2,..."
  default     = ""
}

variable "batch_size" {
  type        = string
  description = "Flush threshold in bytes."
  default     = ""
}

On the Logs with Lambda page in Grafana Cloud Provider, copy the Grafana Cloud Loki write address, and Grafana Cloud Loki username, and your Grafana Cloud access policy token into a Terraform variables file (for example grafana.auto.tfvars).

Warning
Do not commit secrets to version control.

write_address = "@@@https://logs-prod-...grafana.net/loki/api/v1/push@@@" // Grafana Cloud Loki write address
username      = "@@@LOKI_USERNAME@@@" // Grafana Cloud Loki username
password      = "@@@ACCESS_POLICY_TOKEN@@@" // Grafana Cloud access policy token
s3_bucket     = "@@@my-lambda-artifacts@@@"
log_group_names = ["@@@/aws/lambda/my-app@@@"]

Update log_group_names and s3_bucket to the correct bucket names.
Initialize and apply using the following command:
Bash
```
terraform init
terraform apply
```

The snippets above are a minimal starting point. For VPC subnets, security groups, and additional log groups, extend your module or start from the example main.tf in the Lambda Promtail repository.

Explore logs

After you have connected your AWS logs to Grafana Cloud using Lambda Promtail, you can start exploring.

To explore your logs in Grafana Cloud:

On the Logs with Lambda page in Grafana Cloud Provider, click Go to Explore.
In Explore, run a query that selects on __aws_cloudwatch_log_group to confirm events are arriving.

Labels for this workflow

CloudWatch log lines include the following labels: __aws_cloudwatch_log_group, __aws_cloudwatch_owner, and optionally __aws_cloudwatch_log_stream when KEEP_STREAM is true. For the full table and extra labels syntax, refer to Labels in Logs with Lambda.