Documentation Grafana Alloy Reference Components otelcol otelcol.exporter.splunkhec

Community

otelcol.exporter.splunkhec

Community: This component is developed, maintained, and supported by the Alloy user community. Grafana doesn’t offer commercial support for this component. To enable and use community components, you must set the --feature.community-components.enabled flag to true.

otelcol.exporter.splunkhec accepts metrics and traces telemetry data from other otelcol components and sends it to Splunk HEC.

You can specify multiple otelcol.exporter.splunkhec components by giving them different labels.

Usage

alloy Copy

otelcol.exporter.splunkhec "<LABEL>" {
    splunk {
        token = "<YOUR_SPLUNK_TOKEN>"
    }
    client {
        endpoint = "http://splunk.yourdomain.com:8088"
    }
}

Arguments

The otelcol.exporter.splunkhec component doesn’t support any arguments.

Blocks

The following blocks are supported inside the definition of otelcol.exporter.splunkhec:

The > symbol indicates deeper levels of nesting. For example, splunk > batcher refers to a batcher block defined inside a splunk block.

splunk

Required

The splunk block configures Splunk HEC specific settings.

The following arguments are supported:

batcher

heartbeat

otel_to_hec_fields

telemetry

client

Required

The client block configures the HTTP client used by the component.

The following arguments are supported:

debug_metrics

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

disable_high_cardinality_metrics is the Grafana Alloy equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

queue

The queue block configures an in-memory buffer of batches before data is sent to the HTTP server.

The following arguments are supported:

When enabled is true, data is first written to an in-memory buffer before sending it to the configured server. Batches sent to the component’s input exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size.

queue_size determines how long an endpoint outage is tolerated. Assuming 100 requests/second, the default queue size 1000 provides about 10 seconds of outage tolerance. To calculate the correct value for queue_size, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated. A very high value can cause Out Of Memory (OOM) kills.

The num_consumers argument controls how many readers read from the buffer and send data in parallel. Larger values of num_consumers allow data to be sent more quickly at the expense of increased network traffic.

retry_on_failure

The retry_on_failure block configures how failed requests to Splunk HEC are retried.

The following arguments are supported:

When enabled is true, failed batches are retried after a given interval. The initial_interval argument specifies how long to wait before the first retry attempt. If requests continue to fail, the time to wait before retrying increases by the factor specified by the multiplier argument, which must be greater than 1.0. The max_interval argument specifies the upper bound of how long to wait between retries.

The randomization_factor argument is useful for adding jitter between retrying Alloy instances. If randomization_factor is greater than 0, the wait time before retries is multiplied by a random factor in the range [ I - randomization_factor * I, I + randomization_factor * I], where I is the current interval.

If a batch hasn’t been sent successfully, it’s discarded after the time specified by max_elapsed_time elapses. If max_elapsed_time is set to "0s", failed requests are retried forever until they succeed.

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer data for any telemetry signal (metrics, logs, or traces).

Component health

otelcol.exporter.splunkhec is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.exporter.splunkhec doesn’t expose any component-specific debug information.

Example

Open Telemetry Receiver

This example forwards metrics, logs, and traces send to the otelcol.receiver.otlp.default receiver to the Splunk HEC exporter.

alloy Copy

otelcol.receiver.otlp "default" {
    grpc {
        endpoint = "localhost:4317"
    }

    http {
        endpoint               = "localhost:4318"
        compression_algorithms = ["zlib"]
    }

    output {
        metrics = [otelcol.exporter.splunkhec.default.input]
        logs    = [otelcol.exporter.splunkhec.default.input]
        traces  = [otelcol.exporter.splunkhec.default.input]
    }
}

otelcol.exporter.splunkhec "default" {
    client {
        endpoint                = "https://splunkhec.domain.com:8088/services/collector"
        timeout                 = "10s"
        max_idle_conns          = 200
        max_idle_conns_per_host = 200
        idle_conn_timeout       = "10s"
    }

    splunk {
        token              = "SPLUNK_TOKEN"
        source             = "otel"
        sourcetype         = "otel"
        index              = "metrics"
        splunk_app_name    = "OpenTelemetry-Collector Splunk Exporter"
        splunk_app_version = "v0.0.1"

        otel_to_hec_fields {
            severity_text   = "otel.log.severity.text"
            severity_number = "otel.log.severity.number"
        }

        heartbeat {
            interval = "30s"
        }

        telemetry {
            enabled                = true
            override_metrics_names = {
                otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
                otelcol_exporter_splunkhec_heartbeats_sent   = "app_heartbeats_success_total",
            }
            extra_attributes = {
                custom_key   = "custom_value",
                dataset_name = "SplunkCloudBeaverStack",
            }
        }
    }
}

Forward Prometheus Metrics

This example forwards Prometheus metrics from Alloy through a receiver for conversion to Open Telemetry format before finally sending them to Splunk HEC.

alloy Copy

prometheus.exporter.self "default" {
}

prometheus.scrape "metamonitoring" {
  targets    = prometheus.exporter.self.default.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

otelcol.receiver.prometheus "default" {
  output {
    metrics = [otelcol.exporter.splunkhec.default.input]
  }
}


otelcol.exporter.splunkhec "default" {
    splunk {
        token = "SPLUNK_TOKEN"
    }
    client {
        endpoint = "http://splunkhec.domain.com:8088"
    }
}

Forward Loki logs

This example watches for files ending with .log in the path /var/log, tails these logs with Loki and forwards the logs to the configured Splunk HEC endpoint. The Splunk HEC exporter component is setup to send an heartbeat every 5 seconds.

alloy Copy

local.file_match "local_files" {
    path_targets = [{"__path__" = "/var/log/*.log"}]
    sync_period  = "5s"
}

otelcol.receiver.loki "default" {
    output {
        logs = [otelcol.processor.attributes.default.input]
    }
}

otelcol.processor.attributes "default" {
    action {
        key    = "host"
        action = "upsert"
        value  = "myhost"
    }

    action {
        key    = "host.name"
        action = "upsert"
        value  = "myhost"
    }

    output {
        logs = [otelcol.exporter.splunkhec.default.input]
    }
}

loki.source.file "log_scrape" {
    targets       = local.file_match.local_files.targets
    forward_to    = [otelcol.receiver.loki.default.receiver]
    tail_from_end = false
}

otelcol.exporter.splunkhec "default" {
    retry_on_failure {
        enabled = false
    }

    client {
        endpoint                = "http://splunkhec.domain.com:8088"
        timeout                 = "5s"
        max_idle_conns          = 200
        max_idle_conns_per_host = 200
        idle_conn_timeout       = "10s"
        write_buffer_size       = 8000
    }

    sending_queue {
        enabled = false
    }

    splunk {
        token            = "SPLUNK_TOKEN"
        source           = "otel"
        sourcetype       = "otel"
        index            = "devnull"
        log_data_enabled = true

        heartbeat {
            interval = "5s"
        }

        batcher {
            flush_timeout = "200ms"
        }

        telemetry {
            enabled                = true
            override_metrics_names = {
                otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total",
                otelcol_exporter_splunkhec_heartbeats_sent   = "app_heartbeats_success_total",
            }
            extra_attributes = {
                host   = "myhost",
                dataset_name = "SplunkCloudBeaverStack",
            }
        }
    }
}

Compatible components

otelcol.exporter.splunkhec has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer