Collection methods by cloud provider
| Cloud | Method | Best for |
|---|---|---|
| AWS | Firehose | High-volume CloudWatch logs (most use cases) |
| AWS | Lambda agent | Specific log types (ALB, CloudFront, CloudTrail) |
| Azure | Functions | Event-driven from Event Hub |
| Azure | Alloy | Event Hub with flexible processing |
| GCP | Alloy + Pub/Sub | All GCP logs |
Quick decision for AWS
| Scenario | Recommended method |
|---|---|
| High log volumes, cost-sensitive | Firehose |
| CloudWatch logs from apps, RDS, VPC Flow | Firehose |
| ALB access logs, CloudFront, CloudTrail | Lambda agent |
Script
For logs, your collection options look a bit different than metrics.
In AWS, you have two main methods. Amazon Data Firehose is the recommended approach for most CloudWatch logs. It’s a managed streaming service that’s more scalable, about three times cheaper than Lambda, and handles high log volumes seamlessly with built-in retry handling.
Lambda, also called lambda-promtail, is for specific log types that Firehose doesn’t support yet, like ALB access logs, CloudFront logs, and CloudTrail audit logs. Lambda triggers on events to process and forward these logs to Loki.
For Azure, you have similar options: Azure Functions for event-driven collection from Event Hub, or Grafana Alloy for more flexibility.
For GCP, there’s no serverless option. The standard approach is to route logs through Pub/Sub and have Alloy subscribe to that topic.
So your choice depends on your log volumes, cost considerations, and which specific log types you need. Let’s go through each approach.