Documentation Grafana Alloy Reference Components otelcol otelcol.exporter.loadbalancing

otelcol.exporter.loadbalancing

otelcol.exporter.loadbalancing accepts logs and traces from other otelcol components and writes them over the network using the OpenTelemetry Protocol (OTLP) protocol.

You can specify multiple otelcol.exporter.loadbalancing components by giving them different labels.

The decision which backend to use depends on the trace ID or the service name. The backend load doesn’t influence the choice. Even though this load-balancer won’t do round-robin balancing of the batches, the load distribution should be very similar among backends, with a standard deviation under 5% at the current configuration.

otelcol.exporter.loadbalancing is especially useful for backends configured with tail-based samplers which choose a backend based on the view of the full trace.

When a list of backends is updated, some of the signals will be rerouted to different backends. Around R/N of the “routes” will be rerouted differently, where:

• A “route” is either a trace ID or a service name mapped to a certain backend.
• “R” is the total number of routes.
• “N” is the total number of backends.

This should be stable enough for most cases, and the larger the number of backends, the less disruption it should cause.

Usage

alloy Copy

otelcol.exporter.loadbalancing "<LABEL>" {
  resolver {
    ...
  }
  protocol {
    otlp {
      client {}
    }
  }
}

Arguments

You can use the following arguments with otelcol.exporter.loadbalancing:

The routing_key attribute determines how to route signals across endpoints. Its value can be one of the following:

• "service": spans, logs, and metrics with the same service.name will be exported to the same backend.

This is useful when using processors like the span metrics, so all spans for each service are sent to consistent Alloy instances for metric collection. Otherwise, metrics for the same services would be sent to different instances, making aggregations inaccurate.

• "traceID": Spans and logs belonging to the same traceID will be exported to the same backend.
• "resource": Metrics belonging to the same resource will be exported to the same backend.
• "metric": Metrics with the same name will be exported to the same backend.
• "streamID": Metrics with the same streamID will be exported to the same backend.

The loadbalancer configures the exporter for the signal types supported by the routing_key.

The timeout argument is similar to the top-level queue and retry blocks for otelcol.exporter.loadbalancing itself. It helps to re-route data into a new set of healthy backends. This is especially useful for highly elastic environments like Kubernetes, where the list of resolved endpoints changes frequently due to deployments and scaling events.

EXPERIMENTAL: Metrics support in otelcol.exporter.loadbalancing is an experimental feature. Experimental features are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental feature, you must set the stability.level flag to experimental.

Blocks

You can use the following blocks with otelcol.exporter.loadbalancing:

The > symbol indicates deeper levels of nesting. For example, resolver > static refers to a static block defined inside a resolver block.

There are two types of queue and retry blocks:

• The queue and retry blocks under protocol > otlp. This is useful for temporary problems with a specific backend, like transient network issues.
• The top-level queue and retry blocks for otelcol.exporter.loadbalancing. Those configuration options provide capability to re-route data into a new set of healthy backends. This is useful for highly elastic environments like Kubernetes, where the list of resolved endpoints changes frequently due to deployments and scaling events.

resolver

The resolver block configures how to retrieve the endpoint to which this exporter will send data.

Inside the resolver block, either the dns block or the static block should be specified. If both dns and static are specified, dns takes precedence.

aws_cloud_map

The aws_cloud_map block allows users to use otelcol.exporter.loadbalancing when using ECS over EKS in an AWS infrastructure.

The following arguments are supported:

health_status can be set to either of:

• HEALTHY: Only return instances that are healthy.
• UNHEALTHY: Only return instances that are unhealthy.
• ALL: Return all instances, regardless of their health status.
• HEALTHY_OR_ELSE_ALL: Returns healthy instances, unless none are reporting a healthy state. In that case, return all instances. This is also called failing open.

If port isn’t set, a default port defined in CloudMap will be used.

dns

The dns block periodically resolves an IP address via the DNS hostname attribute. This IP address and the port specified via the port attribute will then be used by the gRPC exporter as the endpoint to which to export data to.

The following arguments are supported:

kubernetes

You can use the kubernetes block to load balance across the pods of a Kubernetes service. The Kubernetes API notifies Alloy whenever a new Pod is added or removed from the service. The kubernetes resolver has a much faster response time than the dns resolver because it doesn’t require polling.

The following arguments are supported:

If no namespace is specified inside service, an attempt will be made to infer the namespace for this Alloy. If this fails, the default namespace will be used.

Each of the ports listed in ports will be used with each of the IPs resolved from service.

The “get”, “list”, and “watch” roles must be granted in Kubernetes for the resolver to work.

return_hostnames is useful in certain situations like using Istio in sidecar mode. To use this feature, the service argument must be a headless Service, pointing at a StatefulSet. Also, the service argument must be what’s specified under .spec.serviceName in the StatefulSet.

static

The static block configures a list of endpoints which this exporter will send data to.

The following arguments are supported:

protocol

The protocol block configures protocol-related settings for exporting. At the moment only the OTLP protocol is supported.

otlp

The otlp block configures OTLP-related settings for exporting.

client

The client block configures the gRPC client used by the component. The endpoints used by the client block are the ones from the resolver block

The following arguments are supported:

You can configure an HTTP proxy with the following environment variables:

• HTTPS_PROXY
• NO_PROXY

The HTTPS_PROXY environment variable specifies a URL to use for proxying requests. Connections to the proxy are established via the HTTP CONNECT method.

The NO_PROXY environment variable is an optional list of comma-separated hostnames for which the HTTPS proxy should not be used. Each hostname can be provided as an IP address (1.2.3.4), an IP address in CIDR notation (1.2.3.4/8), a domain name (example.com), or *. A domain name matches that domain and all subdomains. A domain name with a leading “.” (.example.com) matches subdomains only. NO_PROXY is only read when HTTPS_PROXY is set.

Because otelcol.exporter.loadbalancing uses gRPC, the configured proxy server must be able to handle and proxy HTTP/2 traffic.

keepalive

The keepalive block configures keepalive settings for gRPC client connections.

The following arguments are supported:

tls

The tls block configures TLS settings used for the connection to the gRPC server.

tpm

The tpm block configures retrieving the TLS key_file from a trusted device.

The following arguments are supported:

Name	Type	Description	Default	Required
auth	string	The authorization value used to authenticate the TPM device.	""	no
enabled	bool	Load the tls.key_file from TPM.	false	no
owner_auth	string	The owner authorization value used to authenticate the TPM device.	""	no
path	string	Path to the TPM device or Unix domain socket.	""	no

The trusted platform module (TPM) configuration can be used for loading TLS key from TPM. Currently only TSS2 format is supported.

The path attribute is not supported on Windows.

Example

alloy Copy

otelcol.example.component "<LABEL>" {
    ...
    tls {
        ...
        key_file = "my-tss2-key.key"
        tpm {
            enabled = true
            path = "/dev/tpmrm0"
        }
    }
}

In the above example, the private key my-tss2-key.key in TSS2 format will be loaded from the TPM device /dev/tmprm0.

queue

The queue block configures an in-memory buffer of batches before data is sent to the gRPC server.

batch

The batch block configures batching requests based on a timeout and a minimum number of items. By default, the batch block is not used.

retry

The retry block configures how failed requests to the gRPC server are retried.

debug_metrics

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer OTLP-formatted data for telemetry signals of these types:

• logs
• traces

Choose a load balancing strategy

Different Alloy components require different load-balancing strategies. The use of otelcol.exporter.loadbalancing is only necessary for stateful components.

otelcol.processor.tail_sampling

All spans for a given trace ID must go to the same tail sampling Alloy instance.

• This can be done by configuring otelcol.exporter.loadbalancing with routing_key = "traceID".
• If you don’t configure routing_key = "traceID", the sampling decision may be incorrect. The tail sampler must have a full view of the trace when making a sampling decision. For example, a rate_limiting tail sampling strategy may incorrectly pass through more spans than expected if the spans for the same trace are spread out to more than one Alloy instance.

otelcol.connector.spanmetrics

All spans for a given service.name must go to the same spanmetrics Alloy.

• This can be done by configuring otelcol.exporter.loadbalancing with routing_key = "service".
• If you do not configure routing_key = "service", metrics generated from spans might be incorrect. For example, if similar spans for the same service.name end up on different Alloy instances, the two Alloy instances will have identical metric series for calculating span latency, errors, and number of requests. When both Alloy instances attempt to write the metrics to a database such as Mimir, the series may clash with each other. At best, this will lead to an error in Alloy and a rejected write to the metrics database. At worst, it could lead to inaccurate data due to overlapping samples for the metric series.

However, there are ways to scale otelcol.connector.spanmetrics without the need for a load balancer:

1. Each Alloy could add an attribute such as collector.id to make its series unique. Then, for example, you could use a sum by PromQL query to aggregate the metrics from different Alloys. Unfortunately, an extra collector.id attribute has a downside that the metrics stored in the database will have higher cardinality.
2. Spanmetrics could be generated in the backend database instead of in Alloy. For example, span metrics can be generated in Grafana Cloud by the Tempo traces database.

otelcol.connector.servicegraph

It’s challenging to scale otelcol.connector.servicegraph over multiple Alloy instances. For otelcol.connector.servicegraph to work correctly, each “client” span must be paired with a “server” span to calculate metrics such as span duration. If a “client” span goes to one Alloy, but a “server” span goes to another Alloy, then no single Alloy will be able to pair the spans and a metric won’t be generated.

otelcol.exporter.loadbalancing can solve this problem partially if it is configured with routing_key = "traceID". Each Alloy will then be able to calculate a service graph for each “client”/“server” pair in a trace. It’s possible to have a span with similar “server”/“client” values in a different trace, processed by another Alloy. If two different Alloy instances process similar “server”/“client” spans, they will generate the same service graph metric series. If the series from two Alloy are the same, this will lead to issues when writing them to the backend database. You could differentiate the series by adding an attribute such as "collector.id". The series from different Alloys can be aggregated using PromQL queries on the backed metrics database. If the metrics are stored in Grafana Mimir, cardinality issues due to "collector.id" labels can be solved using Adaptive Metrics.

A simpler, more scalable alternative to generating service graph metrics in Alloy is to generate them entirely in the backend database. For example, service graphs can be generated in Grafana Cloud by the Tempo traces database.

Mix stateful components

Different Alloy components may require a different routing_key for otelcol.exporter.loadbalancing. For example, otelcol.processor.tail_sampling requires routing_key = "traceID" whereas otelcol.connector.spanmetrics requires routing_key = "service". To load balance both types of components, two different sets of load balancers have to be set up:

• One set of otelcol.exporter.loadbalancing with routing_key = "traceID", sending spans to Alloys doing tail sampling and no span metrics.
• Another set of otelcol.exporter.loadbalancing with routing_key = "service", sending spans to Alloys doing span metrics and no service graphs.

Unfortunately, this can also lead to side effects. For example, if otelcol.connector.spanmetrics is configured to generate exemplars, the tail sampling Alloys might drop the trace that the exemplar points to. There is no coordination between the tail sampling Alloys and the span metrics Alloys to make sure trace IDs for exemplars are kept.

Component health

otelcol.exporter.loadbalancing is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.exporter.loadbalancing doesn’t expose any component-specific debug information.

Examples

Static resolver

This example accepts OTLP logs and traces over gRPC. It then sends them in a load-balanced way to "localhost:55690" or "localhost:55700".

alloy Copy

otelcol.receiver.otlp "default" {
    grpc {}
    output {
        traces  = [otelcol.exporter.loadbalancing.default.input]
        logs    = [otelcol.exporter.loadbalancing.default.input]
    }
}

otelcol.exporter.loadbalancing "default" {
    resolver {
        static {
            hostnames = ["localhost:55690", "localhost:55700"]
        }
    }
    protocol {
        otlp {
            client {}
        }
    }
}

DNS resolver

When configured with a dns resolver, otelcol.exporter.loadbalancing will do a DNS lookup on regular intervals. Spans are exported to the addresses the DNS lookup returned.

alloy Copy

otelcol.exporter.loadbalancing "default" {
    resolver {
        dns {
            hostname = "alloy-traces-sampling.grafana-cloud-monitoring.svc.cluster.local"
            port     = "34621"
            interval = "5s"
            timeout  = "1s"
        }
    }
    protocol {
        otlp {
            client {}
        }
    }
}

The following example shows a Kubernetes configuration that configures two groups of Alloy instances:

• A pool of load-balancer Alloy instances:
  • Spans are received from instrumented applications via otelcol.receiver.otlp
  • Spans are exported via otelcol.exporter.loadbalancing.
• A pool of sampling Alloy instances:
  • The sampling Alloys run behind a headless service to enable the load-balancer Alloys to discover them.
  • Spans are received from the load-balancer Alloys via otelcol.receiver.otlp
  • Traces are sampled via otelcol.processor.tail_sampling.
  • The traces are exported via otelcol.exporter.otlp to an OTLP-compatible database such as Tempo.

Example Kubernetes configuration

You must fill in the correct OTLP credentials prior to running the example. You can use k3d to start the example:

Bash Copy

k3d cluster create alloy-lb-test
kubectl apply -f kubernetes_config.yaml

To delete the cluster, run:

Bash Copy

k3d cluster delete alloy-lb-test

Kubernetes resolver

When you configure otelcol.exporter.loadbalancing with a kubernetes resolver, the Kubernetes API notifies Alloy whenever a new Pod is added or removed from the service. Spans are exported to the addresses from the Kubernetes API, combined with all the possible ports.

alloy Copy

otelcol.exporter.loadbalancing "default" {
    resolver {
        kubernetes {
            service = "alloy-traces-headless"
            ports   = [ 34621 ]
        }
    }
    protocol {
        otlp {
            client {}
        }
    }
}

The following example shows a Kubernetes configuration that sets up two groups of Alloy instances:

• A pool of load-balancer Alloys:
  • Spans are received from instrumented applications via otelcol.receiver.otlp
  • Spans are exported via otelcol.exporter.loadbalancing.
  • The load-balancer Alloys will get notified by the Kubernetes API any time a Pod is added or removed from the pool of sampling Alloys.
• A pool of sampling Alloyinstances:
  • The sampling Alloy instances don’t need to run behind a headless service.
  • Spans are received from the load-balancer Alloys via otelcol.receiver.otlp
  • Traces are sampled via otelcol.processor.tail_sampling.
  • The traces are exported via otelcol.exporter.otlp to a an OTLP-compatible database such as Tempo.

Example Kubernetes configuration

You must fill in the correct OTLP credentials prior to running the example. You can use k3d to start the example:

Bash Copy

k3d cluster create alloy-lb-test
kubectl apply -f kubernetes_config.yaml

To delete the cluster, run:

Bash Copy

k3d cluster delete alloy-lb-test

Compatible components

otelcol.exporter.loadbalancing has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer