Menu
Grafana Cloud

Configure Application Load Balancer Logs

Send Application Load Balancer access logs to Grafana Cloud Logs using a Lambda function. The following diagram shows the AWS components needed to correctly send logs.

ALB logs processing and sending
ALB logs processing and sending

Complete the following steps to create the Lambda function.

  1. Navigate to your Grafana Cloud portal.
  2. In your Grafana Cloud stack, click Connections in the left-hand menu.
  3. Find and click the AWS tile.
  4. Find and click the Logs with Lambda tile. The Configuration Details page opens.
  5. Click the Configure integration button.

Configure with CloudFormation

Complete the following process to configure with CloudFormation.

Choose method for creating AWS resources

Click Use CloudFormation.

Upload the lambda-promtail zip file to S3

  1. Upload the zip binary to an S3 bucket. The S3 bucket must be in the same AWS region where the Lambda function runs.
  2. Copy and run the the command.

Create a Grafana.com API Token

Create a Grafana.com API token with the necessary permissions to authenticate with Grafana Cloud Logs.

  1. In the API token name box, enter your API token name.
  2. Click Create token to generate the token.
  3. Copy the generated API key.

Launch CloudFormation stack

  1. Click Launch CloudFormation stack.
  2. In AWS, paste the generated API key into the Password field.
  3. In the AccessLogsS3Bucket parameter, specify the S3 bucket where the Application Load Balancer logs are located.

Check for logs

  1. At the Grafana Configuration Details page, click Go to Explore.
  2. Verify logs are being forwarded correctly.

For more details, refer to Explore logs.

Configure with Terraform

Complete the following process to configure with Terraform.

Choose method for creating AWS resources

Click Use Terraform.

Create a Grafana.com API key

  1. In the API token name box, enter your API token name.
  2. Click Create token to generate the token.
  3. Copy the generated API key.

Terraform setup

Configure the AWS CLI. Remember to set the correct AWS region where lambda-promtail should run and pull logs from.

  1. Copy and paste the following snippet into a Terraform file.

    terraform
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4.0"
        }
      }
    }
    
    provider "aws" {
      region = "us-west-2"
    }
    
    resource "aws_s3_object_copy" "lambda_promtail_zipfile" {
      bucket = var.s3_bucket
      key    = var.s3_key
      source = "grafanalabs-cf-templates/lambda-promtail/lambda-promtail.zip"
    }
    
    resource "aws_iam_role" "lambda_promtail_role" {
      name = "GrafanaLabsALBLogsIntegration"
    
      assume_role_policy = jsonencode({
        "Version" : "2012-10-17",
        "Statement" : [
          {
            "Action" : "sts:AssumeRole",
            "Principal" : {
              "Service" : "lambda.amazonaws.com"
            },
            "Effect" : "Allow",
          }
        ]
      })
    }
    
    resource "aws_iam_role_policy" "lambda_promtail_policy_alb_logs" {
      name = "alb-logs"
      role = aws_iam_role.lambda_promtail_role.name
      policy = jsonencode({
        "Statement" : [
          {
            "Action" : [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:PutLogEvents",
            ],
            "Effect" : "Allow",
            "Resource" : "arn:aws:logs:*:*:*",
          },
          {
            "Action" : [
              "s3:GetObject",
            ],
            "Effect" : "Allow",
            "Resource" : format("arn:aws:s3:::%s/*", var.access_logs_s3_bucket),
          }
        ]
      })
    }
    
    resource "aws_lambda_function" "lambda_promtail" {
      function_name = "GrafanaCloudLambdaPromtail"
      role          = aws_iam_role.lambda_promtail_role.arn
    
      timeout     = 60
      memory_size = 128
    
      handler   = "main"
      runtime   = "go1.x"
      s3_bucket = var.s3_bucket
      s3_key    = var.s3_key
    
      environment {
        variables = {
          WRITE_ADDRESS = var.write_address
          USERNAME      = var.username
          PASSWORD      = var.password
          BATCH_SIZE    = var.batch_size
          EXTRA_LABELS  = var.extra_labels
        }
      }
    
      depends_on = [
        aws_s3_object_copy.lambda_promtail_zipfile,
        aws_iam_role_policy.lambda_promtail_policy_alb_logs,
      ]
    }
    
    resource "aws_lambda_function_event_invoke_config" "lambda_promtail_invoke_config" {
      function_name          = aws_lambda_function.lambda_promtail.function_name
      maximum_retry_attempts = 2
    }
    
    // The permission below will allow S3 to trigger lambda-promtail, upon files being added
    resource "aws_lambda_permission" "lambda_promtail_allow_s3" {
      statement_id  = "lambda-promtail-allow-s3"
      action        = "lambda:InvokeFunction"
      function_name = aws_lambda_function.lambda_promtail.function_name
      principal     = "s3.amazonaws.com"
    }
    
    // Configures the access logs S3 bucket to emit a notification to lambda-promtail, when a file
    // is added.
    resource "aws_s3_bucket_notification" "bucket_notification" {
      bucket = var.access_logs_s3_bucket
    
      lambda_function {
        lambda_function_arn = aws_lambda_function.lambda_promtail.arn
        events              = ["s3:ObjectCreated:*"]
      }
    
      depends_on = [
        aws_lambda_permission.lambda_promtail_allow_s3,
      ]
    }
    
    output "role_arn" {
      value       = aws_lambda_function.lambda_promtail.arn
      description = "The ARN of the Lambda function that runs lambda-promtail."
    }
  2. Copy and paste the following snippet into a variables.tf file.

    terraform
    variable "write_address" {
      type        = string
      description = "This is the Grafana Cloud Loki URL that logs will be forwarded to."
      default     = ""
    }
    
    variable "username" {
      type        = string
      description = "The basic auth username for Grafana Cloud Loki."
      default     = ""
    }
    
    variable "password" {
      type        = string
      description = "The basic auth password for Grafana Cloud Loki (your Grafana.com API Key)."
      sensitive   = true
      default     = ""
    }
    
    variable "s3_bucket" {
      type        = string
      description = "The name of the bucket where to upload the 'lambda-promtail.zip' file."
      default     = ""
    }
    
    variable "s3_key" {
      type        = string
      description = "The desired path where to upload the 'lambda-promtail.zip' file (defaults to the root folder)."
      default     = "lambda-promtail.zip"
    }
    
    variable "extra_labels" {
      type        = string
      description = "Comma separated list of extra labels, in the format 'name1,value1,name2,value2,...,nameN,valueN' to add to entries forwarded by lambda-promtail."
      default     = ""
    }
    
    variable "batch_size" {
      type        = string
      description = "Determines when to flush the batch of logs (bytes)."
      default     = ""
    }
    
    variable "access_logs_s3_bucket" {
        type = string
        description = "The S3 buckets where ALB access logs are stored"
        default = ""
    }
  3. Return to the Grafana Configuration Details page.

  4. Copy and paste the write_address, username, and password into the appropriate places in the variables.tf file.

  5. In the variables.tf file, configure variables according to their descriptions. Note that the Access Logs S3 bucket must be in the same regions where the Lambda is deployed.

  6. Run the Terraform apply command:

    bash
     terraform apply -var-file="variables.tf"

    After the Terraform apply command has finished creating the resources, it outputs the role_arn of the Lambda function that runs Lambda-Promtail.

The previous Terraform snippets should get you started with a basic configuration for Lambda-Promtail. For additional setup (specifically, VPC subnets and security groups), refer to this extended example Terraform file, for the comments within the terraform snippet.

Check for logs

  1. At the Grafana Configuration Details page, click Go to Explore.
  2. Verify logs are being forwarded correctly.

For more details, refer to Explore logs.

Explore logs

At the Explore page, use the following query to find all the load balancing access logs:

plain
{__aws_log_type=~"s3_lb"}

Labels

ALB access logs sent to Grafana Cloud Logs include the following special labels assigned to them:

  • __aws_s3_lb: Name of the Application Load Balancer that generated the logs
  • __aws_s3_lb_owner: AWS account ID of the owner of this log

You can specify extra labels (as key-value pairs) to be added to logs streamed by Lambda-Promtail: __extra_<name>=<value>. Add these labels in the CloudFormation parameter ExtraLabels, with the format label1,value1,label2,value2.