Logs with Amazon Data Firehose

You can stream logs from AWS to Grafana Cloud using Cloud Provider Observability and Amazon Data Firehose. Sending AWS logs with Lambda Promtail covers many use cases. However, if you have growing log volumes or have large-scale logs, sending AWS logs with Data Firehose is more scalable and almost three times cheaper.

How it works

Amazon Data Firehose is a fully managed service for delivering real-time streaming data to a destination, such as any custom HTTP endpoint. Data Firehose acts as a stream that allows you to:

Send data into it.
Configure transformations.
Pick a destination where the data is to be delivered (such as an HTTP endpoint, S3 bucket, and so on).

The Data Firehose delivery stream can batch and send ingested records to Grafana Cloud.

You can configure multiple AWS resources to send events to Data Firehose, including:

Amazon CloudWatch logs through subscription filters
VPC flow logs
AWS CloudTrail logs

Other services can also deliver logs data directly to Data Firehose, including AWS Web Application Firewall, Amazon Route 53 Resolver Query Logs, and Amazon VPC Flow.

You can also back up events to an S3 bucket to help prevent data loss for either all log events or only those having a failed delivery and exceeding a specified retry duration. For more information about adjusting your backup based on the amount of log data you’re streaming, refer to Amazon Data Firehose Quota.