Menu
Open source

loki.source.gcplog

loki.source.gcplog retrieves logs from cloud resources such as GCS buckets, load balancers, or Kubernetes clusters running on GCP by making use of Pub/Sub subscriptions.

The component uses either the ‘push’ or ‘pull’ strategy to retrieve log entries and forward them to the list of receivers in forward_to.

Multiple loki.source.gcplog components can be specified by giving them different labels.

Usage

river
loki.source.gcplog "LABEL" {
  pull {
    project_id   = "PROJECT_ID"
    subscription = "SUB_ID"
  }

  forward_to = RECEIVER_LIST
}

Arguments

loki.source.gcplog supports the following arguments:

NameTypeDescriptionDefaultRequired
forward_tolist(LogsReceiver)List of receivers to send log entries to.yes
relabel_rulesRelabelRulesRelabeling rules to apply on log entries.“{}”no

Blocks

The following blocks are supported inside the definition of loki.source.gcplog:

HierarchyNameDescriptionRequired
pullpullConfigures a target to pull logs from a GCP Pub/Sub subscription.no
pushpushConfigures a server to receive logs as GCP Pub/Sub push requests.no
push > httphttpConfigures the HTTP server that receives requests when using the push mode.no
push > grpcgrpcConfigures the gRPC server that receives requests when using the push mode.no

The pull and push inner blocks are mutually exclusive; a component must contain exactly one of the two in its definition. The http and grpc block are just used when the push block is configured.

pull block

The pull block defines which GCP project ID and subscription to read log entries from.

The following arguments can be used to configure the pull block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
project_idstringThe GCP project id the subscription belongs to.yes
subscriptionstringThe subscription to pull logs from.yes
labelsmap(string)Additional labels to associate with incoming logs."{}"no
use_incoming_timestampboolWhether to use the incoming log timestamp.falseno
use_full_lineboolSend the full line from Cloud Logging even if textPayload is available.falseno

To make use of the pull strategy, the GCP project must have been configured to forward its cloud resource logs onto a Pub/Sub topic for loki.source.gcplog to consume.

Typically, the host system also needs to have its GCP credentials configured. One way to do it is to point the GOOGLE_APPLICATION_CREDENTIALS environment variable to the location of a credential configuration JSON file or a service account key.

push block

The push block defines the configuration of the server that receives push requests from GCP’s Pub/Sub servers.

The following arguments can be used to configure the push block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
graceful_shutdown_timeoutdurationTimeout for servers graceful shutdown. If configured, should be greater than zero.“30s”no
push_timeoutdurationSets a maximum processing time for each incoming GCP log entry."0s"no
labelsmap(string)Additional labels to associate with incoming entries."{}"no
use_incoming_timestampboolWhether to use the incoming entry timestamp.falseno
use_full_lineboolSend the full line from Cloud Logging even if textPayload is available. By default, if textPayload is present in the line, then it’s used as log linefalseno

The server listens for POST requests from GCP’s Push subscriptions on HOST:PORT/gcp/api/v1/push.

By default, for both strategies the component assigns the log entry timestamp as the time it was processed, except if use_incoming_timestamp is set to true.

The labels map is applied to every entry that passes through the component.

http

The http block configures the HTTP server.

You can use the following arguments to configure the http block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
conn_limitintMaximum number of simultaneous HTTP connections. Defaults to no limit.0no
listen_addressstringNetwork address on which the server listens for new connections. Defaults to accepting all incoming connections.""no
listen_portintPort number on which the server listens for new connections.8080no
server_idle_timeoutdurationIdle timeout for HTTP server."120s"no
server_read_timeoutdurationRead timeout for HTTP server."30s"no
server_write_timeoutdurationWrite timeout for HTTP server."30s"no

grpc

The grpc block configures the gRPC server.

You can use the following arguments to configure the grpc block. Any omitted fields take their default values.

NameTypeDescriptionDefaultRequired
conn_limitintMaximum number of simultaneous HTTP connections. Defaults to no limit.0no
listen_addressstringNetwork address on which the server listens for new connections. It defaults to accepting all incoming connections.""no
listen_portintPort number on which the server listens for new connections. Defaults to a random free port.0no
max_connection_age_gracedurationAn additive period after max_connection_age after which the connection is forcibly closed."infinity"no
max_connection_agedurationThe duration for the maximum time a connection may exist before it is closed."infinity"no
max_connection_idledurationThe duration after which an idle connection is closed."infinity"no
server_max_concurrent_streamsintLimit on the number of concurrent streams for gRPC calls (0 = unlimited).100no
server_max_recv_msg_sizeintLimit on the size of a gRPC message this server can receive (bytes).4MBno
server_max_send_msg_sizeintLimit on the size of a gRPC message this server can send (bytes).4MBno

Exported fields

loki.source.gcplog does not export any fields.

Component health

loki.source.gcplog is only reported as unhealthy if given an invalid configuration.

Debug information

loki.source.gcplog exposes some debug information per gcplog listener:

  • The configured strategy.
  • Their label set.
  • When using a push strategy, the listen address.

Debug metrics

When using the pull strategy, the component exposes the following debug metrics:

  • loki_source_gcplog_pull_entries_total (counter): Number of entries received by the gcplog target.
  • loki_source_gcplog_pull_parsing_errors_total (counter): Total number of parsing errors while receiving gcplog messages.
  • loki_source_gcplog_pull_last_success_scrape (gauge): Timestamp of target’s last successful poll.

When using the push strategy, the component exposes the following debug metrics:

  • loki_source_gcplog_push_entries_total (counter): Number of entries received by the gcplog target.
  • loki_source_gcplog_push_entries_total (counter): Number of parsing errors while receiving gcplog messages.

Example

This example listens for GCP Pub/Sub PushRequests on 0.0.0.0:8080 and forwards them to a loki.write component.

river
loki.source.gcplog "local" {
  push {}

  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

On the other hand, if we need the server to listen on 0.0.0.0:4040, and forwards them to a loki.write component.

river
loki.source.gcplog "local" {
  push {
    http {
        listen_port = 4040
    }
  }

  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

Compatible components

loki.source.gcplog can accept arguments from the following components:

Note

Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.