Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Open source

loki.source.gcplog

loki.source.gcplog retrieves logs from cloud resources such as GCS buckets, load balancers, or Kubernetes clusters running on GCP by making use of Pub/Sub subscriptions.

The component uses either the ‘push’ or ‘pull’ strategy to retrieve log entries and forward them to the list of receivers in forward_to.

Multiple loki.source.gcplog components can be specified by giving them different labels.

Usage

river
loki.source.gcplog "LABEL" {
  pull {
    project_id   = "PROJECT_ID"
    subscription = "SUB_ID"
  }

  forward_to = RECEIVER_LIST
}

Arguments

loki.source.gcplog supports the following arguments:

NameTypeDescriptionDefaultRequired
forward_tolist(LogsReceiver)List of receivers to send log entries to.yes
relabel_rulesRelabelRulesRelabeling rules to apply on log entries.“{}”no

Blocks

The following blocks are supported inside the definition of loki.source.gcplog:

HierarchyNameDescriptionRequired
pullpullConfigures a target to pull logs from a GCP Pub/Sub subscription.no
pushpushConfigures a server to receive logs as GCP Pub/Sub push requests.no

The pull and push inner blocks are mutually exclusive; a component must contain exactly one of the two in its definition.

pull block

The pull block defines which GCP project ID and subscription to read log entries from.

The following arguments can be used to configure the pull block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
project_idstringThe GCP project id the subscription belongs to.yes
subscriptionstringThe subscription to pull logs from.yes
labelsmap(string)Additional labels to associate with incoming logs."{}"no
use_incoming_timestampboolWhether to use the incoming log timestamp.falseno

To make use of the pull strategy, the GCP project must have been configured to forward its cloud resource logs onto a Pub/Sub topic for loki.source.gcplog to consume.

Typically, the host system also needs to have its GCP credentials configured. One way to do it is to point the GOOGLE_APPLICATION_CREDENTIALS environment variable to the location of a credential configuration JSON file or a service account key.

push block

The push block defines the configuration of the HTTP server that receives push requests from GCP’s Pub/Sub servers.

The following arguments can be used to configure the push block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
http_listen_addressstringThe address the server listens to."0.0.0.0"no
http_listen_portintThe port the server listens to.8080no
push_timeoutdurationSets a maximum processing time for each incoming GCP log entry."0s"no
labelsmap(string)Additional labels to associate with incoming entries."{}"no
use_incoming_timestampboolWhether to use the incoming entry timestamp.falseno

The server listens for POST requests from GCP’s Push subscriptions on HOST:PORT/gcp/api/v1/push.

By default, for both strategies the component assigns the log entry timestamp as the time it was processed, except if use_incoming_timestamp is set to true.

The labels map is applied to every entry that passes through the component.

Exported fields

loki.source.gcplog does not export any fields.

Component health

loki.source.gcplog is only reported as unhealthy if given an invalid configuration.

Debug information

loki.source.gcplog exposes some debug information per gcplog listener:

  • The configured strategy.
  • Their label set.
  • When using a push strategy, the listen address.

Debug metrics

When using the pull strategy, the component exposes the following debug metrics:

  • loki_source_gcplog_pull_entries_total (counter): Number of entries received by the gcplog target.
  • loki_source_gcplog_pull_parsing_errors_total (counter): Total number of parsing errors while receiving gcplog messages.
  • loki_source_gcplog_pull_last_success_scrape (gauge): Timestamp of target’s last successful poll.

When using the push strategy, the component exposes the following debug metrics:

  • loki_source_gcplog_push_entries_total (counter): Number of entries received by the gcplog target.
  • loki_source_gcplog_push_entries_total (counter): Number of parsing errors while receiving gcplog messages.

Example

This example listens for GCP Pub/Sub PushRequests on 0.0.0.0:8080 and forwards them to a loki.write component.

river
loki.source.gcplog "local" {
  push {}

  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}