Documentation Grafana Loki Send data Lambda Promtail Reference

Lambda Promtail reference

This page describes the environment variables, propagated labels, relabeling, and limitations for Lambda Promtail. The Terraform and CloudFormation deployments set most of these values for you. For deployment steps, refer to Deploy with Terraform and Deploy with CloudFormation.

Environment variables

Lambda Promtail reads its configuration from the following environment variables.

Propagated labels

Incoming logs are assigned special labels that you can use in relabeling or in later pipeline stages:

For S3-based logs, <log_type> is one of the following values, which is also used as the value of __aws_log_type:

s3_vpc_flow, s3_lb, s3_cloudtrail, s3_cloudfront, s3_waf, s3_guardduty, s3_msk, s3_access.

For example, an Application Load Balancer log receives the labels __aws_log_type="s3_lb", __aws_s3_lb for the load balancer name, and __aws_s3_lb_owner for the account ID.

Relabeling configuration

Lambda Promtail supports Prometheus-style relabeling through the RELABEL_CONFIGS environment variable. Use relabeling to modify, keep, or drop labels before the function sends logs to Loki. Provide the configuration as a JSON array of relabel rules. Relabeling follows the same principles as Prometheus relabeling. For a detailed explanation, refer to How relabeling in Prometheus works.

Example configurations

Rename a label and capture regular expression groups:

JSON Copy

[
  {
    "source_labels": ["__aws_log_type"],
    "target_label": "log_type",
    "action": "replace",
    "regex": "(.*)",
    "replacement": "${1}"
  }
]

Keep only specific log types:

JSON Copy

[
  {
    "source_labels": ["__aws_log_type"],
    "regex": "s3_.*",
    "action": "keep"
  }
]

Drop internal AWS labels:

JSON Copy

[
  {
    "regex": "__aws_.*",
    "action": "labeldrop"
  }
]

Combine multiple rules:

JSON Copy

[
  {
    "source_labels": ["__aws_log_type"],
    "target_label": "log_type",
    "action": "replace",
    "regex": "(.*)",
    "replacement": "${1}"
  },
  {
    "source_labels": ["__aws_s3_lb"],
    "target_label": "loadbalancer",
    "action": "replace"
  },
  {
    "regex": "__aws_.*",
    "action": "labeldrop"
  }
]

Supported actions

Relabeling supports the same actions as Prometheus:

• replace: Replace a label value with a new value using regular expression capture groups.
• keep: Keep entries where labels match the regular expression.
• drop: Drop entries where labels match the regular expression.
• hashmod: Set a label to the modulus of a hash of labels, which is useful for sharding.
• labelmap: Copy labels to other labels based on regular expression matching.
• labeldrop: Remove labels that match the regular expression.
• labelkeep: Keep only labels that match the regular expression.
• lowercase: Convert label values to lowercase.
• uppercase: Convert label values to uppercase.

Configuration fields

Each relabel rule supports the following fields. All fields are optional except action.

• source_labels: A list of label names to use as input for the action.
• separator: A string that joins source label values. The default is ;.
• target_label: The label to modify. It’s required for the replace and hashmod actions.
• regex: A regular expression to match against. The default is (.+) for most actions.
• replacement: The replacement pattern for the matched regular expression. It supports capture groups such as ${1} and ${2}.
• modulus: The modulus for the hashmod action.
• action: One of the supported actions.

Relabel order and behavior

• Relabeling runs after the function merges the labels from EXTRA_LABELS and drops the labels specified by DROP_LABELS.
• If relabeling removes all labels from an entry, the function drops the entry.
• Rules are processed in order, and each rule can affect the input of later rules.
• Regular expressions in the regex field support full RE2 syntax.
• For the replace action, if the regex doesn’t match, the target label remains unchanged.

Pipeline stages

Set the LOKI_STAGE_CONFIGS environment variable to transform entries before the function forwards them. The value is a JSON array of pipeline stages, where each element maps a stage name to its configuration. Each entry is processed synchronously. If a stage doesn’t finish within PIPELINE_TIMEOUT, the function drops the entry.

Lambda Promtail uses the same log-processing stages as Grafana Alloy. For the available stages and their options, refer to the loki.process component in the Grafana Alloy documentation. The Alloy documentation describes each stage in Alloy syntax. In LOKI_STAGE_CONFIGS, provide the equivalent configuration as JSON.

For example, the following value extracts fields from a JSON log line and then promotes the level field to a label:

JSON Copy

[
  {
    "json": {
      "expressions": {
        "level": "level",
        "message": "msg"
      }
    }
  },
  {
    "labels": {
      "level": ""
    }
  }
]

Example Grafana Alloy configuration

Instead of writing directly to Loki, you can forward logs from Lambda Promtail to a Grafana Alloy collector, which then writes to Loki.

The following Alloy configuration receives logs on the Loki push API endpoint, maps the special __aws_* labels to Loki labels, and forwards the entries to Loki. Set the Lambda Promtail WRITE_ADDRESS to the Alloy endpoint, for example http://<alloy-host>:3500/loki/api/v1/push. In the loki.write component, select the highlighted placeholder to enter your own Loki write endpoint.

Alloy Copy

// Receive logs from Lambda Promtail on the Loki push API endpoint.
loki.source.api "lambda_promtail" {
  http {
    listen_address = "0.0.0.0"
    listen_port    = 3500
  }

  forward_to = [loki.write.default.receiver]

  // Add a static label to indicate that the Lambda Promtail workflow processed these logs.
  labels = {
    source = "lambda-promtail",
  }

  relabel_rules = loki.relabel.lambda_promtail.rules
}

// Map the special __aws_* labels to labels for use in Loki.
loki.relabel "lambda_promtail" {
  forward_to = []

  rule {
    source_labels = ["__aws_log_type"]
    target_label  = "log_type"
  }

  // Map the CloudWatch log group into a label called log_group.
  rule {
    source_labels = ["__aws_cloudwatch_log_group"]
    target_label  = "log_group"
  }

  // Map the load balancer name into a label called loadbalancer_name.
  rule {
    source_labels = ["__aws_s3_lb"]
    target_label  = "loadbalancer_name"
  }
}

// Forward received logs to Loki.
loki.write "default" {
  endpoint {
    url = "http://@@@LOKI_ENDPOINT@@@:3100/loki/api/v1/push"
  }
}

Limitations

The following limitations describe the constraints and trade-offs of running Lambda Promtail, such as how retries and dropped logs are handled, event size caps, and behavior when you forward through a collector. Review them before you deploy to understand where you might lose logs or need to adjust your architecture, and to set expectations for reliability and scale.

Retries and dropped logs

Lambda Promtail applies retries at several layers:

• Sending a batch to the write endpoint: When a batch fails with an HTTP 429, an HTTP 5xx, or a connection-level error, Lambda Promtail retries the send. The retry count is hard-coded to 10 attempts, waiting an exponentially increasing delay between attempts, from 100 milliseconds up to 30 seconds. If every attempt fails, the function drops the batch. Errors other than 429, 5xx, and connection-level errors aren’t retried.
• Lambda invocation: AWS retries the function invocation itself on failure. The provided Terraform sets a maximum of 2 invocation retries with maximum_retry_attempts.
• SQS redrive: If you trigger the function through SQS, a message that fails to process returns to the queue and moves to the dead-letter queue after it reaches the maximum receive count. The provided Terraform sets this count to 5.

CloudWatch event size

Amazon CloudWatch quotas limit the event size to 256 KB. This quota can’t be changed.

Batch behavior when writing to a collector

This limitation is relevant only when Lambda Promtail writes to a collector, such as Grafana Alloy, instead of directly to Loki. Because the collector batches writes to Loki for performance, it can receive a log, return a successful 204 status code, and then be terminated before it writes upstream to Loki. This is rare, but it’s a trade-off of forwarding through a collector.

Availability

For availability, run a set of Grafana Alloy collectors behind a load balancer.

Template and deployment customization

The provided Terraform and CloudFormation files cover the default use cases. More complex deployments, such as adding VPC configuration or subscribing to many CloudWatch log groups, require you to modify and extend the files. The Terraform configuration is more flexible than the CloudFormation templates because it accepts arrays of log group and bucket names and supports VPC configuration.

Collectors between Lambda Promtail and Loki

Forwarding through a collector moves the worst-case stream cardinality from the number of log streams to the number of log groups multiplied by the number of collectors. When you run a set of collectors behind a load balancer, assign each collector a unique label so that logs for the same log group don’t cause out-of-order errors. In Grafana Alloy, add a unique label with the external_labels argument of the loki.write component, for example external_labels = { collector = constants.hostname }. Run a small number of collectors behind a load balancer according to your throughput and redundancy needs.

If you haven’t configured Loki to accept out-of-order writes, the unique label is required.