Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Agent Flow mode Reference Components reference loki.source.awsfirehose

loki.source.awsfirehose

loki.source.awsfirehose receives log entries over HTTP from AWS Firehose and forwards them to other loki.* components.

The HTTP API exposed is compatible with the Firehose HTTP Delivery API. Since the API model that AWS Firehose uses to deliver data over HTTP is generic enough, the same component can be used to receive data from multiple origins:

• AWS CloudWatch logs
• AWS CloudWatch events
• Custom data through DirectPUT requests

The component uses a heuristic to try to decode as much information as possible from each log record, and it falls back to writing the raw records to Loki. The decoding process goes as follows:

• AWS Firehose sends batched requests
• Each record is treated individually
• For each record received in each request:
  • If the record comes from a CloudWatch logs subscription filter, it is decoded and each logging event is written to Loki
  • All other records are written raw to Loki

The component exposes some internal labels, available for relabeling. The following tables describes internal labels available in records coming from any source.

If the source of the Firehose record is CloudWatch logs, the request is further decoded and enriched with even more labels, exposed as follows:

See Examples for a full example configuration showing how to enrich each log entry with these labels.

Usage

river Copy

loki.source.awsfirehose "LABEL" {
    http {
        listen_address = "LISTEN_ADDRESS"
        listen_port = PORT 
    }
    forward_to = RECEIVER_LIST
}

The component will start an HTTP server on the configured port and address with the following endpoints:

• /awsfirehose/api/v1/push - accepting POST requests compatible with AWS Firehose HTTP Specifications.

Arguments

loki.source.awsfirehose supports the following arguments:

The relabel_rules field can make use of the rules export value from a loki.relabel component to apply one or more relabeling rules to log entries before they’re forwarded to the list of receivers in forward_to.

Blocks

The following blocks are supported inside the definition of loki.source.awsfirehose:

http

The http block configures the HTTP server.

The following arguments can be used to configure the http block. Any omitted fields take their default values.

grpc

The grpc block configures the gRPC server.

The following arguments can be used to configure the grpc block. Any omitted fields take their default values.

Exported fields

loki.source.awsfirehose does not export any fields.

Component health

loki.source.awsfirehose is only reported as unhealthy if given an invalid configuration.

Debug metrics

The following are some of the metrics that are exposed when this component is used.

• loki_source_awsfirehose_request_errors (counter): Count of errors while receiving a request.
• loki_source_awsfirehose_record_errors (counter): Count of errors while decoding an individual record.
• loki_source_awsfirehose_records_received (counter): Count of records received.
• loki_source_awsfirehose_batch_size (histogram): Size (in units) of the number of records received per request.

Example

This example starts an HTTP server on 0.0.0.0 address and port 9999. The server receives log entries and forwards them to a loki.write component. The loki.write component will send the logs to the specified loki instance using basic auth credentials provided.

river Copy

loki.write "local" {
    endpoint {
        url = "http://loki:3100/api/v1/push"
        basic_auth {
            username = "<your username>"
            password_file = "<your password file>"
        }
    }
}

loki.source.awsfirehose "loki_fh_receiver" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9999
    }
    forward_to = [
        loki.write.local.receiver,
    ]
}

As another example, if you are receiving records that originated from a CloudWatch logs subscription, you can enrich each received entry by relabeling internal labels. The following configuration builds upon the one above but keeps the origin log stream and group as log_stream and log_group, respectively.

river Copy

loki.write "local" {
    endpoint {
        url = "http://loki:3100/api/v1/push"
        basic_auth {
            username = "<your username>"
            password_file = "<your password file>"
        }
    }
}

loki.source.awsfirehose "loki_fh_receiver" {
    http {
        listen_address = "0.0.0.0"
        listen_port = 9999
    }
    forward_to = [
        loki.write.local.receiver,
    ]
    relabel_rules = loki.relabel.logging_origin.rules
}

loki.relabel "logging_origin" {
  rule {
    action = "replace"
    source_labels = ["__aws_cw_log_group"]
    target_label = "log_group"
  }
  rule {
    action = "replace"
    source_labels = ["__aws_cw_log_stream"]
    target_label = "log_stream"
  }
  forward_to = []
}