Cloud setup GCP logs
This document explain how one can setup Google Cloud Platform to forward its cloud resource logs from a particular GCP project into Google Pubsub topic so that is available for Promtail to consume.
This document assumes, that reader have
gcloud installed and have required permissions(as mentioned in #[Roles and Permission] section)
Roles and Permission
User should have following roles to complete the setup.
Setup Pubsub Topic
Google Pubsub Topic will act as the queue to persist log messages which then can be read from Promtail.
$ gcloud pubsub topics create $TOPIC_ID
$ gcloud pubsub topics create cloud-logs
Setup Log Router
We create a log sink to forward cloud logs into pubsub topic created before
$ gcloud logging sinks create $SINK_NAME $SINK_LOCATION $OPTIONAL_FLAGS
$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \ --log-filter='resource.type=("gcs_bucket")' \ --description="Cloud logs"
Above command also adds
log-filter option which represents what type of logs should get into the destination
For more information on adding
log-filter refer this document
We cover more advanced
Grant log sink the pubsub publisher role
Find the writer identity service account of the log sink just created:
gcloud logging sinks describe \ --format='value(writerIdentity)' $SINK_NAME
gcloud logging sinks describe \ --format='value(writerIdentity)' cloud-logs
Create an IAM policy binding to allow log sink to publish messages to the topic:
gcloud pubsub topics add-iam-policy-binding $TOPIC_ID \ --member=$WRITER_IDENTITY --role=roles/pubsub.publisher
gcloud pubsub topics add-iam-policy-binding cloud-logs \ --member=serviceAccount:email@example.com --role=roles/pubsub.publisher
Create Pubsub subscription for Grafana Loki
We create subscription for the pubsub topic we create above and Promtail uses this subscription to consume log messages.
$ gcloud pubsub subscriptions create cloud-logs --topic=$TOPIC_ID \ --ack-deadline=$ACK_DEADLINE \ --message-retention-duration=$RETENTION_DURATION \
$ gcloud pubsub subscriptions create cloud-logs --topic=projects/my-project/topics/cloud-logs \ --ack-deadline=10 \ --message-retention-duration=7d
For more fine grained options, refer to the
gcloud pubsub subscriptions --help
ServiceAccount for Promtail
We need a service account with following permissions.
This enables Promtail to read log entries from the pubsub subscription created before.
you can find example for Promtail scrape config for
If you are scraping logs from multiple GCP projects, then this serviceaccount should have above permissions in all the projects you are tyring to scrape.
Sometimes you may wish to clear the pending pubsub queue containing logs.
These messages stays in Pubsub Subscription until they’re acknowledged. The following command removes log messages without needing to be consumed via Promtail or any other pubsub consumer.
gcloud pubsub subscriptions seek <subscription-path> --time=<yyyy-mm-ddThh:mm:ss>
To delete all the old messages until now, set
--time to current time.
gcloud pubsub subscriptions seek projects/my-project/subscriptions/cloud-logs --time=$(date +%Y-%m-%dT%H:%M:%S)
Advanced log filter
So far we’ve covered admitting GCS bucket logs into Grafana Loki, but often one may need to add multiple cloud resource logs and may also need to exclude unnecessary logs. The following is a more complex example.
We use the
log-filter option to include logs and the
exclusion option to exclude them.
Include following cloud resource logs
- GCS bucket
- HTTP Load balancer
And we exclude specific HTTP load balancer logs based on payload and status code.
$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \ --log-filter='resource.type=("gcs_bucket OR k8s_cluster OR service_account OR iam_role OR api OR audited_resource OR http_load_balancer")' \ --description="Cloud logs" \ --exclusion='name=http_load_balancer,filter=<<EOF resource.type="http_load_balancer" ( ( jsonPayload.statusDetails=("byte_range_caching" OR "websocket_closed") ) OR ( http_request.status=(101 OR 206) ) ) EOF
Related Enterprise Logs resources
Grafana Enterprise Logs: Logging with security and scale
Join us for this webinar, which will cover: Challenges with logging as organizations scale and the volume of logs explodes, how Grafana Enterprise Logs enables organizations to make logs available to any team members who need them, features available in GEL and how to get access, a live product demo so you can see GEL for the first time
VIDEO: Watch this first-look demo of the new Grafana Enterprise Logs
Based on Loki, Grafana Enterprise Logs is part of the Grafana Enterprise Stack for composing and scaling observability on your own infrastructure.
Introducing Grafana Enterprise Logs, a core part of the Grafana Enterprise Stack integrated observability solution
Powered by the Loki open source project, the Enterprise Logs offering joins metrics and dashboards in our enterprise-ready stack for self-managed observability.