GCP Logs integration for Grafana Cloud
Move your logs from GCP to Grafana Cloud using GCP Pub/Sub and the Grafana Agent
Are you interested in a version of this integration which allows you to skip setting up the agent? Consider filling out our form to be contacted in the future for Beta Access.
Before you begin
The GCP Logs integration is based on configuring the Grafana Agent to pull logs from a GCP Pub/Sub topic which contains your logs from GCP Cloud Monitoring. The logs are routed from cloud monitoring to pub/sub via customizable logging sink.
You will need to have the gcloud CLI
installed and ensure you have roles pubsub.editor
, and logging.configWriter
to complete this setup
Setup Pubsub Topic
Google Pubsub Topic will act as the queue to persist log messages which then can be read from the agent.
$ gcloud pubsub topics create $TOPIC_ID
e.g:
$ gcloud pubsub topics create cloud-logs
Setup Log Sink
Next we create log sink to forward logs into pubsub topic created previously
$ gcloud logging sinks create $SINK_NAME $SINK_LOCATION $OPTIONAL_FLAGS
e.g:
$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \
--log-filter='resource.type=("gcs_bucket")' \
--description="Cloud logs"
Above command also adds log-filter
option which represents what type of logs should get into the destination pubsub
topic.
For more information on adding log-filter
refer this document
Grant log sink the pubsub publisher role
Find the writer identity service account of the log sink just created:
gcloud logging sinks describe \
--format='value(writerIdentity)' $SINK_NAME
For example:
gcloud logging sinks describe \
--format='value(writerIdentity)' cloud-logs
Create an IAM policy binding to allow log sink to publish messages to the topic:
gcloud pubsub topics add-iam-policy-binding $TOPIC_ID \
--member=$WRITER_IDENTITY --role=roles/pubsub.publisher
For example:
gcloud pubsub topics add-iam-policy-binding cloud-logs \
--member=serviceAccount:pxxxxxxxxx-xxxxxx@gcp-sa-logging.iam.gserviceaccount.com --role=roles/pubsub.publisher
Create Pubsub subscription for Log Delivery
A subscription will be created for the pubsub topic configured above and the agent will use this subscription to consume log messages.
$ gcloud pubsub subscriptions create cloud-logs --topic=$TOPIC_ID \
--ack-deadline=$ACK_DEADLINE \
--message-retention-duration=$RETENTION_DURATION \
e.g:
$ gcloud pubsub subscriptions create cloud-logs --topic=projects/my-project/topics/cloud-logs \
--ack-deadline=10 \
--message-retention-duration=7d
For more fine grained options, refer to the gcloud pubsub subscriptions --help
Setup using Terraform
You also have the option of creating the required resources for GCP Log collection with terraform.
How to use Terraform is outside the scope of this guide. You can find this tutorial on how to work with Terraform and GCP useful.
// Provider module
provider "google" {
project = "$GCP_PROJECT_ID"
}
// Topic
resource "google_pubsub_topic" "main" {
name = "cloud-logs"
}
// Log sink
variable "inclusion_filter" {
type = string
description = "Optional GCP Logs query which can filter logs being routed to the pub/sub topic and promtail"
}
resource "google_logging_project_sink" "main" {
name = "cloud-logs"
destination = "pubsub.googleapis.com/${google_pubsub_topic.main.id}"
filter = var.inclusion_filter
unique_writer_identity = true
}
resource "google_pubsub_topic_iam_binding" "log-writer" {
topic = google_pubsub_topic.main.name
role = "roles/pubsub.publisher"
members = [
google_logging_project_sink.main.writer_identity,
]
}
// Subscription
resource "google_pubsub_subscription" "main" {
name = "cloud-logs"
topic = google_pubsub_topic.main.name
}
Then, to create the new resources run the snippet below after filling in the required variables.
terraform apply \
-var="inclusion_filter=<GCP Logs query of what logs to include>"
ServiceAccount for the agent
The agent will need to run with a service account that has the pubsub.subscriber
permission.
This will enable the agent to read log entries from the pubsub subscription created before.
Install GCP Logs integration for Grafana Cloud
- In your Grafana Cloud stack, click Connections in the left-hand menu.
- Find GCP Logs and click its tile to open the integration.
- Review the prerequisites in the Configuration Details tab and set up Grafana Agent to send GCP Logs metrics and logs to your Grafana Cloud instance.
- Click Install to add this integration’s to your Grafana Cloud instance, and you can start monitoring your GCP Logs setup.
Post-install configuration for the GCP Logs integration
In the agent configuration file, find the placeholder <gcp_project_id>
and <gcp_pubsub_subscription_name>
names and
replace them with the GCP project id and subscription name which was created in the pre-install steps.
Advanced configuration
The integration configuration found at logs.configs.scrape_configs
can be modified further for more advanced usages. Some examples
of those advanced scenarios are below.
Multiple Consumers Single Subscription
- gcplog:
labels:
job: integration/gcp
project_id: project-1
subscription: subscription-1
subscription_type: pull
use_incoming_timestamp: false
job_name: integrations/gcp
relabel_configs:
- action: replace
source_labels:
- __gcp_logname
target_label: logname
- action: replace
source_labels:
- __gcp_resource_type
target_label: resource_type
- gcplog:
labels:
job: integration/gcp
project_id: project-1
subscription: subscription-1
subscription_type: pull
use_incoming_timestamp: false
job_name: integrations/gcp-2
relabel_configs:
- action: replace
source_labels:
- __gcp_logname
target_label: logname
- action: replace
source_labels:
- __gcp_resource_type
target_label: resource_type
Multiple Subscriptions
- gcplog:
labels:
job: integration/gcp
project_id: project-1
subscription: subscription-1
subscription_type: pull
use_incoming_timestamp: false
job_name: integrations/gcp
relabel_configs:
- action: replace
source_labels:
- __gcp_logname
target_label: logname
- action: replace
source_labels:
- __gcp_resource_type
target_label: resource_type
- gcplog:
labels:
job: integration/gcp
project_id: project-1
subscription: subscription-2
subscription_type: pull
use_incoming_timestamp: false
job_name: integrations/gcp-2
relabel_configs:
- action: replace
source_labels:
- __gcp_logname
target_label: logname
- action: replace
source_labels:
- __gcp_resource_type
target_label: resource_type
Configuration snippets for Grafana Agent
Below logs.configs.scrape_configs
, insert the following lines according to your environment.
- gcplog:
subscription_type: pull
project_id: <gcp_project_id>
subscription: <gcp_pubsub_subscription_name>
labels:
job: integration/gcp
use_incoming_timestamp: false
job_name: integrations/gcp
relabel_configs:
- action: replace
source_labels:
- __gcp_logname
target_label: logname
- action: replace
source_labels:
- __gcp_resource_type
target_label: resource_type
Full example configuration for Grafana Agent
Refer to the following Grafana Agent configuration for a complete example that contains all the snippets used for the GCP Logs integration. This example also includes metrics that are sent to monitor your Grafana Agent instance.
integrations:
prometheus_remote_write:
- basic_auth:
password: <your_prom_pass>
username: <your_prom_user>
url: <your_prom_url>
agent:
enabled: true
relabel_configs:
- action: replace
source_labels:
- agent_hostname
target_label: instance
- action: replace
target_label: job
replacement: "integrations/agent-check"
metric_relabel_configs:
- action: keep
regex: (prometheus_target_.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
source_labels:
- __name__
# Add here any snippet that belongs to the `integrations` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
logs:
configs:
- clients:
- basic_auth:
password: <your_loki_pass>
username: <your_loki_user>
url: <your_loki_url>
name: integrations
positions:
filename: /tmp/positions.yaml
scrape_configs:
# Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
- gcplog:
subscription_type: pull
project_id: <gcp_project_id>
subscription: <gcp_pubsub_subscription_name>
labels:
job: integration/gcp
use_incoming_timestamp: false
job_name: integrations/gcp
relabel_configs:
- action: replace
source_labels:
- __gcp_logname
target_label: logname
- action: replace
source_labels:
- __gcp_resource_type
target_label: resource_type
metrics:
configs:
- name: integrations
remote_write:
- basic_auth:
password: <your_prom_pass>
username: <your_prom_user>
url: <your_prom_url>
scrape_configs:
# Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
# For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
global:
scrape_interval: 60s
wal_directory: /tmp/grafana-agent-wal
Changelog
# 0.0.2 - August 2023
* Add regex filter for logs datasource
# 0.0.1 - December 2022
* Initial Release
Cost
By connecting your GCP Logs instance to Grafana Cloud, you might incur charges. To view information on the number of active series that your Grafana Cloud account uses for metrics included in each Cloud tier, see Active series and dpm usage and Cloud tier pricing.
Related resources from Grafana Labs


