Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

DocumentationGrafana AgentFlow modeReferenceComponents referenceotelcol.exporter.loadbalancing

otelcol.exporter.loadbalancing

BETA: This is a beta component. Beta components are subject to breaking changes, and may be replaced with equivalent functionality that cover the same use case.

otelcol.exporter.loadbalancing accepts logs and traces from other otelcol components and writes them over the network using the OpenTelemetry Protocol (OTLP) protocol.

NOTE: otelcol.exporter.loadbalancing is a wrapper over the upstream OpenTelemetry Collector loadbalancing exporter. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

Multiple otelcol.exporter.loadbalancing components can be specified by giving them different labels.

The decision which backend to use depends on the trace ID or the service name. The backend load doesn’t influence the choice. Even though this load-balancer won’t do round-robin balancing of the batches, the load distribution should be very similar among backends, with a standard deviation under 5% at the current configuration.

otelcol.exporter.loadbalancing is especially useful for backends configured with tail-based samplers which choose a backend based on the view of the full trace.

When a list of backends is updated, some of the signals will be rerouted to different backends. Around R/N of the “routes” will be rerouted differently, where:

• A “route” is either a trace ID or a service name mapped to a certain backend.
• “R” is the total number of routes.
• “N” is the total number of backends.

This should be stable enough for most cases, and the larger the number of backends, the less disruption it should cause.

Usage

river Copy

otelcol.exporter.loadbalancing "LABEL" {
  resolver {
    ...
  }
  protocol {
    otlp {
      client {}
    }
  }
}

Arguments

otelcol.exporter.loadbalancing supports the following arguments:

The routing_key attribute determines how to route signals across endpoints. Its value could be one of the following:

• "service": spans with the same service.name will be exported to the same backend. This is useful when using processors like the span metrics, so all spans for each service are sent to consistent Agent instances for metric collection. Otherwise, metrics for the same services would be sent to different Agents, making aggregations inaccurate.
• "traceID": spans belonging to the same traceID will be exported to the same backend.

Blocks

The following blocks are supported inside the definition of otelcol.exporter.loadbalancing:

The > symbol indicates deeper levels of nesting. For example, resolver > static refers to a static block defined inside a resolver block.

resolver block

The resolver block configures how to retrieve the endpoint to which this exporter will send data.

Inside the resolver block, either the dns block or the static block should be specified. If both dns and static are specified, dns takes precedence.

static block

The static block configures a list of endpoints which this exporter will send data to.

The following arguments are supported:

dns block

The dns block periodically resolves an IP address via the DNS hostname attribute. This IP address and the port specified via the port attribute will then be used by the gRPC exporter as the endpoint to which to export data to.

The following arguments are supported:

kubernetes block

You can use the kubernetes block to load balance across the pods of a Kubernetes service. The Agent will be notified by the Kubernetes API whenever a new pod is added or removed from the service.

The following arguments are supported:

If no namespace is specified inside service, an attempt will be made to infer the namespace for this Agent. If this fails, the default namespace will be used.

Each of the ports listed in ports will be used with each of the IPs resolved from service.

The “get”, “list”, and “watch” roles must be granted in Kubernetes for the resolver to work.

protocol block

The protocol block configures protocol-related settings for exporting. At the moment only the OTLP protocol is supported.

otlp block

The otlp block configures OTLP-related settings for exporting.

client block

The client block configures the gRPC client used by the component. The endpoints used by the client block are the ones from the resolver block

The following arguments are supported:

By default, requests are compressed with gzip. The compression argument controls which compression mechanism to use. Supported strings are:

• "gzip"
• "zlib"
• "deflate"
• "snappy"
• "zstd"

If compression is set to "none" or an empty string "", no compression is used.

The supported values for balancer_name are listed in the gRPC documentation on Load balancing:

• pick_first: Tries to connect to the first address, uses it for all RPCs if it connects, or tries the next address if it fails (and keeps doing that until one connection is successful). Because of this, all the RPCs will be sent to the same backend.
• round_robin: Connects to all the addresses it sees and sends an RPC to each backend one at a time in order. For example, the first RPC is sent to backend-1, the second RPC is sent to backend-2, and the third RPC is sent to backend-1.

The :authority header in gRPC specifies the host to which the request is being sent. It’s similar to the Host header in HTTP requests. By default, the value for :authority is derived from the endpoint URL used for the gRPC call. Overriding :authority could be useful when routing traffic using a proxy like Envoy, which makes routing decisions based on the value of the :authority header.

You can configure an HTTP proxy with the following environment variables:

• HTTPS_PROXY
• NO_PROXY

The HTTPS_PROXY environment variable specifies a URL to use for proxying requests. Connections to the proxy are established via the HTTP CONNECT method.

The NO_PROXY environment variable is an optional list of comma-separated hostnames for which the HTTPS proxy should not be used. Each hostname can be provided as an IP address (1.2.3.4), an IP address in CIDR notation (1.2.3.4/8), a domain name (example.com), or *. A domain name matches that domain and all subdomains. A domain name with a leading “.” (.example.com) matches subdomains only. NO_PROXY is only read when HTTPS_PROXY is set.

Because otelcol.exporter.loadbalancing uses gRPC, the configured proxy server must be able to handle and proxy HTTP/2 traffic.

tls block

The tls block configures TLS settings used for the connection to the gRPC server.

The following arguments are supported:

If the server doesn’t support TLS, you must set the insecure argument to true.

To disable tls for connections to the server, set the insecure argument to true.

If reload_interval is set to "0s", the certificate never reloaded.

The following pairs of arguments are mutually exclusive and can’t both be set simultaneously:

• ca_pem and ca_file
• cert_pem and cert_file
• key_pem and key_file

keepalive block

The keepalive block configures keepalive settings for gRPC client connections.

The following arguments are supported:

queue block

The queue block configures an in-memory buffer of batches before data is sent to the gRPC server.

The following arguments are supported:

When enabled is true, data is first written to an in-memory buffer before sending it to the configured server. Batches sent to the component’s input exported field are added to the buffer as long as the number of unsent batches doesn’t exceed the configured queue_size.

queue_size determines how long an endpoint outage is tolerated. Assuming 100 requests/second, the default queue size 5000 provides about 50 seconds of outage tolerance. To calculate the correct value for queue_size, multiply the average number of outgoing requests per second by the time in seconds that outages are tolerated.

The num_consumers argument controls how many readers read from the buffer and send data in parallel. Larger values of num_consumers allow data to be sent more quickly at the expense of increased network traffic.

retry block

The retry block configures how failed requests to the gRPC server are retried.

The following arguments are supported:

When enabled is true, failed batches are retried after a given interval. The initial_interval argument specifies how long to wait before the first retry attempt. If requests continue to fail, the time to wait before retrying increases by the factor specified by the multiplier argument, which must be greater than 1.0. The max_interval argument specifies the upper bound of how long to wait between retries.

The randomization_factor argument is useful for adding jitter between retrying agents. If randomization_factor is greater than 0, the wait time before retries is multiplied by a random factor in the range [ I - randomization_factor * I, I + randomization_factor * I], where I is the current interval.

If a batch hasn’t been sent successfully, it is discarded after the time specified by max_elapsed_time elapses. If max_elapsed_time is set to "0s", failed requests are retried forever until they succeed.

debug_metrics block

The debug_metrics block configures the metrics that this component generates to monitor its state.

The following arguments are supported:

disable_high_cardinality_metrics is the Grafana Agent equivalent to the telemetry.disableHighCardinalityMetrics feature gate in the OpenTelemetry Collector. It removes attributes that could cause high cardinality metrics. For example, attributes with IP addresses and port numbers in metrics about HTTP and gRPC connections are removed.

Exported fields

The following fields are exported and can be referenced by other components:

input accepts otelcol.Consumer OTLP-formatted data for telemetry signals of these types:

• logs
• traces

Component health

otelcol.exporter.loadbalancing is only reported as unhealthy if given an invalid configuration.

Debug information

otelcol.exporter.loadbalancing does not expose any component-specific debug information.

Example

This example accepts OTLP logs and traces over gRPC. It then sends them in a load-balanced way to “localhost:55690” or “localhost:55700”.

river Copy

otelcol.receiver.otlp "default" {
    grpc {}
    output {
        traces  = [otelcol.exporter.loadbalancing.default.input]
        logs    = [otelcol.exporter.loadbalancing.default.input]
    }
}

otelcol.exporter.loadbalancing "default" {
    resolver {
        static {
            hostnames = ["localhost:55690", "localhost:55700"]
        }
    }
    protocol {
        otlp {
            client {}
        }
    }
}

Compatible components

otelcol.exporter.loadbalancing has exports that can be consumed by the following components:

• Components that consume OpenTelemetry otelcol.Consumer