otelcol.receiver.filelog
Public preview: This is a public preview component. Public preview components are subject to breaking changes, and may be replaced with equivalent functionality that cover the same use case. The
stability.level
flag must be set topublic-preview
or below to use the component.
otelcol.receiver.filelog
reads log entries from files and forwards them to other otelcol.*
components.
Note
otelcol.receiver.filelog
is a wrapper over the upstream OpenTelemetry Collectorfilelog
receiver. Bug reports or feature requests will be redirected to the upstream repository, if necessary.
You can specify multiple otelcol.receiver.filelog
components by giving them different labels.
Usage
otelcol.receiver.filelog "<LABEL>" {
include = [...]
output {
logs = [...]
}
}
Arguments
You can use the following arguments with otelcol.receiver.filelog
:
encoding
must be one of utf-8
, utf-16le
, utf-16be
, ascii
, big5
, or nop
.
Refer to the upstream receiver documentation for more details.
start_at
must be one of beginning
or end
. The header
block may only be used if start_at
is beginning
.
compression
must be either `` or gzip
.
operators
The operators
list is a list of stanza operators that transform the log entries after they have been read.
For example, if container logs are being collected you may want to utilize the stanza container
parser operator to add relevant attributes to the log entries.
otelcol.receiver.filelog "default" {
...
operators = [
{
type = "container"
}
]
}
Blocks
The following blocks are supported inside the definition of
otelcol.receiver.filelog
:
The > symbol indicates deeper levels of nesting.
For example, ordering_criteria
> sort_by
refers to a sort_by
block defined inside a ordering_criteria
block.
output
Required
The output
block configures a set of components to forward resulting telemetry data to.
The following arguments are supported:
You must specify the output
block, but all its arguments are optional.
By default, telemetry data is dropped.
Configure the metrics
, logs
, and traces
arguments accordingly to send telemetry data to other components.
debug_metrics
The debug_metrics
block configures the metrics that this component generates to monitor its state.
The following arguments are supported:
disable_high_cardinality_metrics
is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics
feature gate in the OpenTelemetry Collector.
It removes attributes that could cause high cardinality metrics.
For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.
Note
If configured,
disable_high_cardinality_metrics
only applies tootelcol.exporter.*
andotelcol.receiver.*
components.
level
is the Alloy equivalent to the telemetry.metrics.level
feature gate in the OpenTelemetry Collector.
Possible values are "none"
, "basic"
, "normal"
and "detailed"
.
header
The header
block configures logic for parsing a log header line into additional attributes added to each log entry.
It may only be used when start_at
is set to beginning
.
The following arguments are supported:
If a header
block is not set, no log lines will be treated as header metadata.
The metadata_operators
list is a list of stanza operators that parses metadata from the header.
Any attributes created from the embedded operators pipeline will be applied to all log entries in the file.
For example, you might use a regex_parser
to process a header line that has been identified by the pattern
expression.
The following example shows a fictitious header line, and then the header
block that would parse an environment
attribute from it.
HEADER_IDENTIFIER env="production"
...
otelcol.receiver.filelog "default" {
...
header {
pattern = '^HEADER_IDENTIFIER .*$'
metadata_operators = [
{
type = "regex_parser"
regex = 'env="(?P<environment>.+)"'
}
]
}
}
multiline
The multiline
block configures logic for splitting incoming log entries.
The following arguments are supported:
A multiline
block must contain either line_start_pattern
or line_end_pattern
.
If a multiline
block isn’t set, log entries will not be split.
ordering_criteria
The ordering_criteria
block configures the order in which log files discovered will be processed.
The following arguments are supported:
sort_by
The sort_by
repeatable block configures the way the fields parsed in the ordering_criteria
block will be applied to sort the discovered log files.
The following arguments are supported:
sort_type
must be one of numeric
, lexicographic
, timestamp
, or mtime
.
When using numeric
, lexicographic
, or timestamp
sort_type
, a named capture group defined in the regex
attribute in ordering_criteria
must be provided in regex_key
.
When using mtime
sort_type
, the file’s modified time will be used to sort.
The location
and layout
arguments are only applicable when sort_type
is timestamp
.
The location
argument specifies a Time Zone identifier. The available locations depend on the local IANA Time Zone database.
Refer to the list of tz database time zones in Wikipedia for a non-comprehensive list.
retry_on_failure
The retry_on_failure
block configures the retry behavior when the receiver encounters an error downstream in the pipeline.
A backoff algorithm is used to delay the retry upon subsequent failures.
The following arguments are supported:
If max_elapsed_time
is set to 0
data is never discarded.
Exported fields
otelcol.receiver.filelog
doesn’t export any fields.
Component health
otelcol.receiver.filelog
is only reported as unhealthy if given an invalid configuration.
Debug metrics
otelcol.receiver.filelog
doesn’t expose any component-specific debug metrics.
Example
This example reads log entries using the otelcol.receiver.filelog
receiver and they are logged by a otelcol.exporter.debug
component.
It expects the logs to start with an ISO8601 compatible timestamp and parses it from the log using the regex_parser
operator.
otelcol.receiver.filelog "default" {
include = ["/var/log/*.log"]
operators = [{
type = "regex_parser",
regex = "^(?P<timestamp>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3,6}Z)",
timestamp = {
parse_from = "attributes.timestamp",
layout = "%Y-%m-%dT%H:%M:%S.%fZ",
location = "UTC",
},
}]
output {
logs = [otelcol.exporter.debug.default.input]
}
}
otelcol.exporter.debug "default" {}
Compatible components
otelcol.receiver.filelog
can accept arguments from the following components:
- Components that export OpenTelemetry
otelcol.Consumer
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.