Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Instrument and send databreadcrumb arrow Grafana Alloybreadcrumb arrow Referencebreadcrumb arrow Componentsbreadcrumb arrow otelcolbreadcrumb arrow otelcol.storage.file
Public preview Open source Grafana Cloud

otelcol.storage.file

Public preview: This is a public preview component. Public preview components are subject to breaking changes, and may be replaced with equivalent functionality that cover the same use case. To enable and use a public preview component, you must set the stability.level flag to public-preview or below.

otelcol.storage.file exposes a handler that other otelcol components can use to write state to a local directory. The current implementation of this component uses bbolt to store and read data on disk.

Note

otelcol.storage.file is a wrapper over the upstream OpenTelemetry Collector filestorage extension. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

You can specify multiple otelcol.storage.file components by giving them different labels.

Usage

Alloy 
otelcol.storage.file "<LABEL>" {
}

Arguments

You can use the following arguments with otelcol.storage.file:

NameTypeDescriptionDefaultRequired
create_directoryboolWill the component be responsible for creating the directory.trueno
directorystringThe path to the dedicated data storage directory.no
directory_permissionsstringThe octal file permissions used when creating the directory if create_directory is set."0750"no
fsyncboolWill fsync be called after each write operation.falseno
timeoutdurationThe timeout for file storage operations."1s"no

The default directory used for file storage is a subdirectory of the data-alloy directory located in the Alloy working directory. This will vary depending on the path specified by the command line flag --storage-path.

create_directory is false by default in the upstream receiver. create_directory is true by default in Alloy because directory defaults to the data-alloy path in Alloy and that directory is owned by the Alloy process.

Blocks

You can use the following blocks with otelcol.storage.file:

BlockDescriptionRequired
compactionConfigures file storage compaction.no
debug_metricsConfigures the metrics that this component generates to monitor its state.no

compaction

The compaction block defines the compaction parameters for the file storage.

NameTypeDescriptionDefaultRequired
check_intervaldurationThe interval to check if online compaction is required."5s"no
cleanup_on_startboolCleanup temporary files on component start.falseno
directorystringThe path to the directory where temporary compaction artifacts will be stored.no
max_transaction_sizeintMaximum number of items present in a single compaction iteration.65536no
on_reboundboolRun compaction online when rebound conditions are met.falseno
on_startboolRun compaction on component start.falseno
rebound_needed_threshold_mibintFile storage total allocated size boundary to mark need for online compaction.100no
rebound_trigger_threshold_mibintFile storage used allocated size boundary to trigger online compaction.10no

The default directory used for file storage is a subdirectory of the data-alloy directory located in the Alloy working directory. This will vary depending on the path specified by the command line flag --storage-path.

If on_rebound online compaction is enabled, compaction will be triggered when total allocated data is greater than rebound_needed_threshold_mib and used allocated data is less than rebound_trigger_threshold_mib. More detailed information about the way the component supports file compaction for allocated disk storage recovery can be found in the upstream component’s documentation.

debug_metrics

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

NameTypeDescriptionDefaultRequired
disable_high_cardinality_metricsbooleanWhether to disable certain high cardinality metrics.trueno

disable_high_cardinality_metrics is the Alloy equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

Note

If configured, disable_high_cardinality_metrics only applies to otelcol.exporter.* and otelcol.receiver.* components.

Exported fields

The following fields are exported and can be referenced by other components:

NameTypeDescription
handlercapsule(otelcol.Handler)A value that other components can use to persist state to file storage.

Component health

otelcol.storage.file is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.storage.file doesn’t expose any component-specific debug information.

Examples

otelcol.receiver.filelog

This examples uses an otelcol.storage.file component to store file offsets for an otelcol.receiver.filelog component. This will only use a small amount of data for each file that has been read so it’s unlikely that you will need to be concerned about compaction settings.

The default settings of the component will place the bbolt file for the receiver in <STORAGE_PATH>/otelcol.storage.file.default/receiver_filelog_default

Alloy 

otelcol.storage.file "default" {}

otelcol.receiver.filelog "default" {
    include = ["/var/log/*.log"]
    storage = otelcol.storage.file.default.handler
    operators = [{
      type = "regex_parser",
      regex = "^(?P<timestamp>[^ ]+)",
      timestamp = {
        parse_from = "attributes.timestamp",
        layout = "%Y-%m-%dT%H:%M:%S.%fZ",
        location = "UTC",
      },
    }]
    output {
        logs = [otelcol.exporter.debug.default.input]
    }
}


otelcol.exporter.debug "default" {}