Experimental
otelcol.exporter.awss3
EXPERIMENTAL: This is an experimental component. Experimental components are subject to frequent breaking changes, and may be removed with no equivalent replacement. The
stability.level
flag must be set toexperimental
to use the component.
otelcol.exporter.awss3
accepts telemetry data from other otelcol
components and writes them to an AWS S3 bucket.
Note
otelcol.exporter.awss3
is a wrapper over the upstream OpenTelemetry Collector Contribawss3
exporter. Bug reports or feature requests will be redirected to the upstream repository if necessary.
You can specify multiple otelcol.exporter.awss3
components by giving them different labels.
Usage
otelcol.exporter.awss3 "<LABEL>" {
s3_uploader {
region = "<REGION>"
s3_bucket = "<BUCKET_NAME>"
s3_prefix = "<PREFIX>"
}
}
Replace the following:
<LABEL>
: The label for theotelcol.exporter.awss3
component.<REGION>
: The AWS region.<BUCKET_NAME>
: The S3 bucket.<PREFIX>
: The prefix for the S3 key.
Arguments
otelcol.exporter.awss3
supports no arguments and is configured completely through inner blocks.
Blocks
The following blocks are supported inside the definition of
otelcol.exporter.awss3
:
s3_uploader block
The s3_uploader
block configures the AWS S3 bucket details used by the component.
The following arguments are supported:
marshaler block
Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:
otlp_json
(default): the OpenTelemetry Protocol format, represented as JSON.otlp_proto
: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.sumo_ic
: the Sumo Logic Installed Collector Archive format. This format is supported only for logs.body
: export the log body as string. This format is supported only for logs.
The following arguments are supported:
debug_metrics block
The debug_metrics
block configures the metrics that this component generates to monitor its state.
The following arguments are supported:
disable_high_cardinality_metrics
is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics
feature gate in the OpenTelemetry Collector.
It removes attributes that could cause high cardinality metrics.
For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.
Note
If configured,
disable_high_cardinality_metrics
only applies tootelcol.exporter.*
andotelcol.receiver.*
components.
level
is the Alloy equivalent to the telemetry.metrics.level
feature gate in the OpenTelemetry Collector.
Possible values are "none"
, "basic"
, "normal"
and "detailed"
.
sending_queue block
The sending_queue
block configures an in-memory buffer of batches before data is sent to s3.
The following arguments are supported:
When enabled
is true
, data is first written to an in-memory buffer before sending it to the configured server.
Batches sent to the component’s input
exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size
.
queue_size
determines how long an endpoint outage is tolerated.
Assuming 100 requests/second, the default queue size 1000
provides about 10 seconds of outage tolerance.
To calculate the correct value for queue_size
, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated. A very high value can cause Out Of Memory (OOM) kills.
The num_consumers
argument controls how many readers read from the buffer and send data in parallel.
Larger values of num_consumers
allow data to be sent more quickly at the expense of increased network traffic.
Compression
none
(default): File compression isn’t used.gzip
: Files are compressed with Gzip. This doesn’t supportsumo_ic
marshaler.
Exported fields
The following fields are exported and can be referenced by other components:
input
accepts otelcol.Consumer
data for any telemetry signal (metrics,
logs, or traces).
Component health
otelcol.exporter.awss3
is only reported as unhealthy if given an invalid
configuration.
Debug information
otelcol.exporter.awss3
doesn’t expose any component-specific debug
information.
Debug metrics
otelcol_exporter_sent_spans_total
(counter): Number of spans successfully sent to destination.otelcol_exporter_send_failed_spans_total
(counter): Number of spans in failed attempts to send to destination.otelcol_exporter_queue_capacity
(gauge): Fixed capacity of the retry queue (in batches).otelcol_exporter_queue_size
(gauge): Current size of the retry queue (in batches).rpc_client_duration_milliseconds
(histogram): Measures the duration of inbound RPC.rpc_client_request_size_bytes
(histogram): Measures size of RPC request messages (uncompressed).rpc_client_requests_per_rpc
(histogram): Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.rpc_client_response_size_bytes
(histogram): Measures size of RPC response messages (uncompressed).rpc_client_responses_per_rpc
(histogram): Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
Example
This example forwards scrape logs to an AWS S3 Bucket:
local.file_match "logs" {
path_targets = [{
__address__ = "localhost",
__path__ = "/var/log/{syslog,messages,*.log}",
instance = constants.hostname,
job = "integrations/node_exporter",
}]
}
loki.source.file "logs" {
targets = local.file_match.logs.targets
forward_to = [otelcol.receiver.loki.default.receiver]
}
otelcol.receiver.loki "default" {
output {
logs = [otelcol.exporter.awss3.logs.input]
}
}
otelcol.exporter.awss3 "logs" {
s3_uploader {
region = "us-east-1"
s3_bucket = "logs_bucket"
s3_prefix = "logs"
}
}
Compatible components
otelcol.exporter.awss3
has exports that can be consumed by the following components:
- Components that consume OpenTelemetry
otelcol.Consumer
Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.