loki.source.gcplog
loki.source.gcplog
retrieves logs from cloud resources such as GCS buckets, load balancers, or Kubernetes clusters running on GCP by making use of Pub/Sub subscriptions.
The component uses either the ‘push’ or ‘pull’ strategy to retrieve log entries and forward them to the list of receivers in forward_to
.
You can specify multiple loki.source.gcplog
components by giving them different labels.
Usage
loki.source.gcplog "<LABEL>" {
pull {
project_id = "<PROJECT_ID>"
subscription = "<SUB_ID>"
}
forward_to = <RECEIVER_LIST>
}
Arguments
loki.source.gcplog
supports the following arguments:
Blocks
You can use the following blocks with loki.source.gcplog
:
The > symbol indicates deeper levels of nesting.
For example, push
> grpc
refers to a grpc
block defined inside a push
block.
The pull
and push
inner blocks are mutually exclusive.
A component must contain exactly one of the two in its definition.
The http
and grpc
block are just used when the push
block is configured.
pull
The pull
block defines which GCP project ID and subscription to read log entries from.
The following arguments can be used to configure the pull
block.
Any omitted fields take their default values.
To make use of the pull
strategy, the GCP project must have been configured to forward its cloud resource logs onto a Pub/Sub topic for loki.source.gcplog
to consume.
Typically, the host system also needs to have its GCP credentials configured.
One way to do it, is to point the GOOGLE_APPLICATION_CREDENTIALS
environment variable to the location of a credential configuration JSON file or a service account key.
push
The push
block defines the configuration of the server that receives push requests from the GCP Pub/Sub servers.
The following arguments can be used to configure the push
block.
Any omitted fields take their default values.
The server listens for POST requests from GCP Push subscriptions on HOST:PORT/gcp/api/v1/push
.
By default, for both strategies the component assigns the log entry timestamp as the time it was processed, except if use_incoming_timestamp
is set to true.
The labels
map is applied to every entry that passes through the component.
grpc
The grpc
block configures the gRPC server.
You can use the following arguments to configure the grpc
block. Any omitted fields take their default values.
http
The http
block configures the HTTP server.
You can use the following arguments to configure the http
block. Any omitted fields take their default values.
Exported fields
loki.source.gcplog
doesn’t export any fields.
Component health
loki.source.gcplog
is only reported as unhealthy if given an invalid configuration.
Debug information
loki.source.gcplog
exposes some debug information per gcplog listener:
- The configured strategy.
- Their label set.
- When using a
push
strategy, the listen address.
Debug metrics
When using the pull
strategy, the component exposes the following debug metrics:
loki_source_gcplog_pull_entries_total
(counter): Number of entries received by the gcplog target.loki_source_gcplog_pull_last_success_scrape
(gauge): Timestamp of target’s last successful poll.loki_source_gcplog_pull_parsing_errors_total
(counter): Total number of parsing errors while receiving gcplog messages.
When using the push
strategy, the component exposes the following debug metrics:
loki_source_gcplog_push_entries_total
(counter): Number of entries received by the gcplog target.loki_source_gcplog_push_entries_total
(counter): Number of parsing errors while receiving gcplog messages.
Example
This example listens for GCP Pub/Sub PushRequests on 0.0.0.0:8080
and forwards them to a loki.write
component.
loki.source.gcplog "local" {
push {}
forward_to = [loki.write.local.receiver]
}
loki.write "local" {
endpoint {
url = "loki:3100/api/v1/push"
}
}
This example listens for GCP Pub/Sub PushRequests on 0.0.0.0:4040
, and forwards them to a loki.write
component.
loki.source.gcplog "local" {
push {
http {
listen_port = 4040
}
}
forward_to = [loki.write.local.receiver]
}
loki.write "local" {
endpoint {
url = "loki:3100/api/v1/push"
}
}
Compatible components
loki.source.gcplog
can accept arguments from the following components:
- Components that export Loki
LogsReceiver
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.