Note: This feature is only available in v1.1.0 release candidates. For the latest release candidates, refer to Download Grafana Enterprise Logs.
The Grafana Enterprise Logs (GEL) gateway is a service target. It can proxy requests to other GEL microservices. You can also use it for client-side load balancing of requests proxied to the distributors.
There is more than one way to configure the gateway: you can configure it via a code block in the YAML configuration file for GEL, or you can use command-line flags.
Configuration code block
The gateway has its own configuration block in the GEL configuration files:
gateway: proxy: default: <backend_proxy_config> [ admin_api: <backend_proxy_config> ] [ compactor: <backend_proxy_config> ] [ distributor: <backend_proxy_config> ] [ ingester: <backend_proxy_config> ] [ query_frontend: <backend_proxy_config> ] [ ruler: <backend_proxy_config> ]
To configure the gateway via command-line flags, make sure that each flag contains the path to the equivalent configuration field.
To configure the distributor backend proxy URL, use the flag
Likewise, given a configuration file field path of
gateway.proxy.query_frontend.url, the flag is
Note: A path is joined by a period (
.) and dashes (
-). Underscores (
_) are invalid within these command-line flags.
backend_proxy section specifies the URL and configuration of the backend that you want proxied.
A URL is only required for the
default proxy configuration. All other proxy configurations use the
default URL if it is not explicitly configured.
Other proxy configurations do not inherit any other configuration from the
default proxy configuration.
url: <url> | default = <gateway.proxy.default.url> [ enable_keepalive: <boolean> | default = true ] [ tls_enabled: <boolean> | default = false ] [ tls_ca_path: <string> | default = "" ] [ tls_cert_path: <string> | default = "" ] [ tls_key_path: <string> | default = "" ] [ tls_insecure_skip_verify: <boolean> | default = false ] [ read_timeout: <duration> | default = "120s" ] [ write_timeout: <duration> | default = "30s" ]
Client-side load balancing
If you use a backend proxy URL that beings with
dns:///, the gateway creates a gRPC proxy with client-side round-robin load balancing instead of the default HTTP reverse proxy.
To configure client-side load balancing for requests to the distributors only, set the
Note: There are three
/characters in the preceding DNS URL, which means that you are using the default DNS authority. For details about DNS URLs, refer to RFC 4501.
Client-side load balancing is useful in ensuring that distributors are evenly loaded with requests. Prometheus remote-write clients use HTTP persistent connections, also known as HTTP keep-alive, the clients re-use a single TCP connection for multiple requests and responses, which results in reduced latency for subsequent requests.
Kubernetes Services are not load balancers; initial TCP connections are made using a random endpoint. After the connection is established, the same remote-write client talks to the same distributor server for its lifetime. This can mean an uneven load for your distributors and worse cluster performance overall.
The GEL gateway solves this problem by exposing an HTTP server for receiving the client requests while using gRPC to talk to the distributors. The gRPC proxy maintains a list of endpoints that are returned from the DNS lookup, and keeps persistent connections to each endpoint. The proxies are also configured to perform per-request, client-side load balancing across the endpoints.
The Gateway exposes the ring endpoints for the rulers, distributors, ingesters and compactors. These
admin scope permissions.
/ringfor the ingesters
/ruler/ringfor the rulers
/compactor/ringfor the compactors
/distributor/ringfor the distributors