Instrument and send data

Grafana Alloy

Reference

Components

otelcol

otelcol.exporter.splunkhec

Open source Grafana Cloud Community

`otelcol.exporter.splunkhec`

Community: This component is developed, maintained, and supported by the Alloy user community. Grafana doesn’t offer commercial support for this component. To enable and use community components, you must set the --feature.community-components.enabled flag to true.

otelcol.exporter.splunkhec accepts metrics and traces telemetry data from other otelcol components and sends it to Splunk HEC.

Note
otelcol.exporter.splunkhec is a wrapper over the upstream OpenTelemetry Collector splunkhec exporter. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

You can specify multiple otelcol.exporter.splunkhec components by giving them different labels.

Usage

otelcol.exporter.splunkhec "<LABEL>" {
    splunk {
        token = "<YOUR_SPLUNK_TOKEN>"
    }
    client {
        endpoint = "http://splunk.yourdomain.com:8088"
    }
}

Arguments

The otelcol.exporter.splunkhec component doesn’t support any arguments. You can configure this component with blocks.

Blocks

You can use the following blocks with otelcol.exporter.splunkhec:

Block	Description	Required
`splunk`	Configures the Splunk HEC exporter.	yes
`splunk` > `batcher`	(Deprecated) Configures batching requests based on a timeout and a minimum number of items.	no
`splunk` > `heartbeat`	Configures the exporters heartbeat settings.	no
`splunk` > `otel_to_hec_fields`	Configures mapping of OpenTelemetry to HEC Fields.	no
`splunk` > `telemetry`	Configures the exporters telemetry.	no
`client`	Configures the HTTP client used to send data to Splunk HEC.	yes
`debug_metrics`	Configures the metrics that this component generates to monitor its state.	no
`otel_attrs_to_hec_metadata`	Configures mapping of resource attributes to HEC metadata fields.	no
`sending_queue`	Configures batching of data before sending.	no
`sending_queue` > `batch`	Configures batching requests based on a timeout and a minimum number of items.	no
`retry_on_failure`	Configures retry mechanism for failed requests.	no

The > symbol indicates deeper levels of nesting. For example, splunk > batcher refers to a batcher block defined inside a splunk block.

`splunk`

Required

The splunk block configures Splunk HEC specific settings.

The following arguments are supported:

Name	Type	Description	Default	Required
`token`	`secret`	Splunk HEC Token.		yes
`disable_compression`	`bool`	Disable Gzip compression.	`false`	no
`export_raw`	`bool`	Send only the logs body when targeting HEC raw endpoint.	`false`	no
`health_check_enabled`	`bool`	Used to verify Splunk HEC health on exporter startup.	`true`	no
`health_path`	`string`	Path for the health API.	`"/services/collector/health"`	no
`index`	`string`	Splunk index name.	`""`	no
`log_data_enabled`	`bool`	Enable sending logs from the exporter. One of `log_data_enabled` or `profiling_data_enabled` must be `true`.	`true`	no
`max_content_length_logs`	`uint`	Maximum log payload size in bytes. Must be less than 838860800 (~800MB).	`2097152`	no
`max_content_length_metrics`	`uint`	Maximum metric payload size in bytes. Must be less than 838860800 (~800MB).	`2097152`	no
`max_content_length_traces`	`uint`	Maximum trace payload size in bytes. Must be less than 838860800 (~800MB).	`2097152`	no
`max_event_size`	`uint`	Maximum event payload size in bytes. Must be less than 838860800 (~800MB).	`5242880`	no
`profiling_data_enabled`	`bool`	Enable sending profiling data from the exporter. One of `log_data_enabled` or `profiling_data_enabled` must be `true`.	`true`	no
`source_type`	`string`	Splunk source type.	`""`	no
`source`	`string`	Splunk source.	`""`	no
`splunk_app_name`	`string`	Used to track telemetry for Splunk Apps by name.	`"Alloy"`	no
`splunk_app_version`	`string`	Used to track telemetry by App version.	`""`	no
`use_multi_metrics_format`	`bool`	Use multi-metrics format to save space during ingestion.	`false`	no

`batcher`

Warning
The batcher block is deprecated and will be removed in a future release. Use the sending_queue > batch block instead.

Name	Type	Description	Default	Required
`enabled`	`bool`	Whether to not enqueue batches before sending to the consumerSender.	`false`	no
`flush_timeout`	`duration`	The time after which a batch will be sent regardless of its size.	`"200ms"`	no
`max_size`	`uint`	The maximum size of a batch. If the batch exceeds this value, it’s broken up into smaller batches. Must be greater than or equal to `min_size`. Set this value to zero to disable the maximum size limit.	`0`	no
`min_size`	`uint`	The minimum size of a batch.	`8192`	no
`sizer`	`string`	The unit of measure for the batch size. Must be one of `items`, `bytes`, or `requests`.	`"items"`	no

`heartbeat`

Name	Type	Description	Default	Required
`interval`	`duration`	Time interval for the heartbeat interval, in seconds.	`"0s"`	no
`startup`	`bool`	Send heartbeat events on exporter startup.	`false`	no

`otel_to_hec_fields`

Name	Type	Description	Default	Required
`severity_number`	`string`	Maps severity number field to a specific HEC field.	`""`	no
`severity_text`	`string`	Maps severity text field to a specific HEC field.	`""`	no

`telemetry`

Name	Type	Description	Default	Required
`enabled`	`bool`	Enable telemetry inside the exporter.	`false`	no
`override_metrics_names`	`map(string)`	Override metrics for internal metrics in the exporter.		no

`client`

Required

The client block configures the HTTP client used by the component.

The following arguments are supported:

Name	Type	Description	Default	Required
`endpoint`	`string`	The Splunk HEC endpoint to use.		yes
`disable_keep_alives`	`bool`	Disable HTTP keep-alive.	`false`	no
`idle_conn_timeout`	`duration`	Time to wait before an idle connection closes itself.	`"45s"`	no
`insecure_skip_verify`	`bool`	Ignores insecure server TLS certificates.	`false`	no
`max_conns_per_host`	`int`	Limits the total (dialing,active, and idle) number of connections per host. Zero means no limit	`0`	no
`max_idle_conns_per_host`	`int`	Limits the number of idle HTTP connections the host can keep open.	`0`	no
`max_idle_conns`	`int`	Limits the number of idle HTTP connections the client can keep open.	`100`	no
`read_buffer_size`	`int`	Size of the read buffer the HTTP client uses for reading server responses.	`0`	no
`timeout`	`duration`	Time to wait before marking a request as failed.	`"15s"`	no
`write_buffer_size`	`int`	Size of the write buffer the HTTP client uses for writing requests.	`0`	no

`otel_attrs_to_hec_metadata`

The otel_attrs_to_hec_metadata block configures the mapping of resource attributes to HEC metadata fields. This allows resource attributes like host.name to be mapped to the top-level host field in the HEC JSON payload.

The following arguments are supported:

Name	Type	Description	Default	Required
`host`	`string`	Specifies the mapping of a specific unified model attribute value to the standard host field of a HEC event.	`"host.name"`	no
`index`	`string`	Specifies the mapping of a specific unified model attribute value to the standard index field of a HEC event.	`"com.splunk.index"`	no
`source`	`string`	Specifies the mapping of a specific unified model attribute value to the standard source field of a HEC event.	`"com.splunk.source"`	no
`sourcetype`	`string`	Specifies the mapping of a specific unified model attribute value to the standard sourcetype field of a HEC event.	`"com.splunk.sourcetype"`	no

`debug_metrics`

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

Name	Type	Description	Default	Required
`disable_high_cardinality_metrics`	`boolean`	Whether to disable certain high cardinality metrics.	`true`	no

disable_high_cardinality_metrics is the Alloy equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

Note
If configured, disable_high_cardinality_metrics only applies to otelcol.exporter.* and otelcol.receiver.* components.

`sending_queue`

The sending_queue block configures an in-memory buffer of batches before data is sent to the HTTP server.

The following arguments are supported:

Name	Type	Description	Default	Required
`block_on_overflow`	`boolean`	The behavior when the component’s `TotalSize` limit is reached.	`false`	no
`enabled`	`boolean`	Enables a buffer before sending data to the client.	`true`	no
`num_consumers`	`number`	Number of readers to send batches written to the queue in parallel.	`10`	no
`queue_size`	`number`	Maximum number of unwritten batches allowed in the queue at the same time.	`1000`	no
`sizer`	`string`	How the queue and batching is measured.	`"requests"`	no
`wait_for_result`	`boolean`	Determines if incoming requests are blocked until the request is processed or not.	`false`	no
`storage`	`capsule(otelcol.Handler)`	Handler from an `otelcol.storage` component to use to enable a persistent queue mechanism.		no

The blocking argument is deprecated in favor of the block_on_overflow argument.

When block_on_overflow is true, the component will wait for space. Otherwise, operations will immediately return a retryable error.

When enabled is true, data is first written to an in-memory buffer before sending it to the configured server. Batches sent to the component’s input exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size.

queue_size determines how long an endpoint outage is tolerated. Assuming 100 requests/second, the default queue size 1000 provides about 10 seconds of outage tolerance. To calculate the correct value for queue_size, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated. A very high value can cause Out Of Memory (OOM) kills.

The sizer argument could be set to:

requests: number of incoming batches of metrics, logs, traces (the most performant option).
items: number of the smallest parts of each signal (spans, metric data points, log records).
bytes: the size of serialized data in bytes (the least performant option).

The num_consumers argument controls how many readers read from the buffer and send data in parallel. Larger values of num_consumers allow data to be sent more quickly at the expense of increased network traffic.

If an otelcol.storage.* component is configured and provided in the queue’s storage argument, the queue uses the provided storage extension to provide a persistent queue and the queue is no longer stored in memory. Any data persisted will be processed on startup if Alloy is killed or restarted. Refer to the exporterhelper documentation in the OpenTelemetry Collector repository for more details.

`batch`

The batch block configures batching requests based on a timeout and a minimum number of items. By default, the batch block is not used.

The following arguments are supported:

Name	Type	Description	Required
`flush_timeout`	`duration`	Time after which a batch will be sent regardless of its size. Must be a non-zero value.	yes
`min_size`	`number`	The minimum size of a batch.	yes
`max_size`	`number`	The maximum size of a batch, enables batch splitting.	yes
`sizer`	`string`	How the queue and batching is measured. Overrides the sizer set at the `sending_queue` level for batching.	yes

max_size must be greater than or equal to min_size.

The sizer argument can be set to:

items: The number of the smallest parts of each span, metric data point, or log record.
bytes: the size of serialized data in bytes (the least performant option).

`retry_on_failure`

The retry_on_failure block configures how failed requests to Splunk HEC are retried.

The following arguments are supported:

Name	Type	Description	Default	Required
`enabled`	`boolean`	Enables retrying failed requests.	`true`	no
`initial_interval`	`duration`	Initial time to wait before retrying a failed request.	`"5s"`	no
`max_elapsed_time`	`duration`	Maximum time to wait before discarding a failed batch.	`"5m"`	no
`max_interval`	`duration`	Maximum time to wait between retries.	`"30s"`	no
`multiplier`	`number`	Factor to grow wait time before retrying.	`1.5`	no
`randomization_factor`	`number`	Factor to randomize wait time before retrying.	`0.5`	no

When enabled is true, failed batches are retried after a given interval. The initial_interval argument specifies how long to wait before the first retry attempt. If requests continue to fail, the time to wait before retrying increases by the factor specified by the multiplier argument, which must be greater than 1.0. The max_interval argument specifies the upper bound of how long to wait between retries.

The randomization_factor argument is useful for adding jitter between retrying Alloy instances. If randomization_factor is greater than 0, the wait time before retries is multiplied by a random factor in the range [ I - randomization_factor * I, I + randomization_factor * I], where I is the current interval.

If a batch hasn’t been sent successfully, it’s discarded after the time specified by max_elapsed_time elapses. If max_elapsed_time is set to "0s", failed requests are retried forever until they succeed.

Exported fields

The following fields are exported and can be referenced by other components:

Name	Type	Description
`input`	`otelcol.Consumer`	A value other components can use to send telemetry data to.

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.exporter.splunkhec is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.exporter.splunkhec doesn’t expose any component-specific debug information.

Example

OpenTelemetry Receiver

This example forwards metrics, logs, and traces send to the otelcol.receiver.otlp.default receiver to the Splunk HEC exporter.

otelcol.receiver.otlp "default" {
    grpc {
        endpoint = "localhost:4317"
    }

    http {
        endpoint               = "localhost:4318"
        compression_algorithms = ["zlib"]
    }

    output {
        metrics = [otelcol.exporter.splunkhec.default.input]
        logs    = [otelcol.exporter.splunkhec.default.input]
        traces  = [otelcol.exporter.splunkhec.default.input]
    }
}

otelcol.exporter.splunkhec "default" {
    client {
        endpoint                = "https://splunkhec.domain.com:8088/services/collector"
        timeout                 = "10s"
        max_idle_conns          = 200
        max_idle_conns_per_host = 200
        idle_conn_timeout       = "10s"
    }

    splunk {
        token              = "SPLUNK_TOKEN"
        source             = "otel"
        sourcetype         = "otel"
        index              = "metrics"
        splunk_app_name    = "OpenTelemetry-Collector Splunk Exporter"
        splunk_app_version = "v0.0.1"

        otel_to_hec_fields {
            severity_text   = "otel.log.severity.text"
            severity_number = "otel.log.severity.number"
        }

        heartbeat {
            interval = "30s"
        }

        telemetry {
            enabled                = true
            override_metrics_names = {
                otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
                otelcol_exporter_splunkhec_heartbeats_sent   = "app_heartbeats_success_total",
            }
            extra_attributes = {
                custom_key   = "custom_value",
                dataset_name = "SplunkCloudBeaverStack",
            }
        }
    }
}

Forward Prometheus Metrics

This example forwards Prometheus metrics from Alloy through a receiver for conversion to OpenTelemetry format before finally sending them to Splunk HEC.

prometheus.exporter.self "default" {
}

prometheus.scrape "metamonitoring" {
  targets    = prometheus.exporter.self.default.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

otelcol.receiver.prometheus "default" {
  output {
    metrics = [otelcol.exporter.splunkhec.default.input]
  }
}


otelcol.exporter.splunkhec "default" {
    splunk {
        token = "SPLUNK_TOKEN"
    }
    client {
        endpoint = "http://splunkhec.domain.com:8088"
    }
}

Forward Loki logs

This example watches for files ending with .log in the path /var/log, tails these logs with Loki and forwards the logs to the configured Splunk HEC endpoint. The Splunk HEC exporter component is setup to send an heartbeat every 5 seconds.

local.file_match "local_files" {
    path_targets = [{"__path__" = "/var/log/*.log"}]
    sync_period  = "5s"
}

otelcol.receiver.loki "default" {
    output {
        logs = [otelcol.processor.resourcedetection.default.input]
    }
}

otelcol.processor.resourcedetection "default" {
    detectors = ["system"]

    system {
        hostname_sources = ["os", "dns"]
        resource_attributes {
            host.name {
                enabled = true
            }
        }
    }

    output {
        logs = [otelcol.exporter.splunkhec.default.input]
    }
}

loki.source.file "log_scrape" {
    targets       = local.file_match.local_files.targets
    forward_to    = [otelcol.receiver.loki.default.receiver]
    tail_from_end = false
}

otelcol.exporter.splunkhec "default" {
    retry_on_failure {
        enabled = false
    }

    client {
        endpoint                = "http://splunkhec.domain.com:8088"
        timeout                 = "5s"
        max_idle_conns          = 200
        max_idle_conns_per_host = 200
        idle_conn_timeout       = "10s"
        write_buffer_size       = 8000
    }

    sending_queue {
        enabled = false
    }

    // Configure mapping of resource attributes to HEC metadata fields
    otel_attrs_to_hec_metadata {
        host       = "host.name"           // Maps host.name resource attribute to top-level host field
        source     = "com.splunk.source"   // Maps com.splunk.source attribute to source field
        sourcetype = "com.splunk.sourcetype" // Maps com.splunk.sourcetype attribute to sourcetype field
        index      = "com.splunk.index"    // Maps com.splunk.index attribute to index field
    }

    splunk {
        token            = "SPLUNK_TOKEN"
        source           = "otel"
        sourcetype       = "otel"
        index            = "devnull"
        log_data_enabled = true

        heartbeat {
            interval = "5s"
        }

        batcher {
            flush_timeout = "200ms"
        }

        telemetry {
            enabled                = true
            override_metrics_names = {
                otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
                otelcol_exporter_splunkhec_heartbeats_sent   = "app_heartbeats_success_total",
            }
            extra_attributes = {
                host   = "myhost",
                dataset_name = "SplunkCloudBeaverStack",
            }
        }
    }
}

Compatible components

otelcol.exporter.splunkhec has exports that can be consumed by the following components:

Components that consume OpenTelemetry otelcol.Consumer

Note
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. Refer to the linked documentation for more details.

Was this page helpful?

Email docs@grafana.com

Help and support

Community

otelcol.exporter.splunkhec

Usage

Arguments

Blocks

splunk

batcher

heartbeat

otel_to_hec_fields

telemetry

client

otel_attrs_to_hec_metadata

debug_metrics

sending_queue

batch

retry_on_failure

Exported fields

Component health

Debug information

Example

OpenTelemetry Receiver

Forward Prometheus Metrics

Forward Loki logs

Compatible components

Was this page helpful?

Related resources from Grafana Labs

`otelcol.exporter.splunkhec`

`splunk`

`batcher`

`heartbeat`

`otel_to_hec_fields`

`telemetry`

`client`

`otel_attrs_to_hec_metadata`

`debug_metrics`

`sending_queue`

`batch`

`retry_on_failure`