The Grafana Enterprise Traces (GET) gateway is a service target. It can proxy requests to other GET microservices. You can also use it for client-side load balancing of requests proxied to the distributors.
There is more than one way to configure the gateway: you can configure it via a code block in the YAML configuration file for GET, or you can use command-line flags.
Configuration code block
The gateway has its own configuration block in the GET configuration files:
gateway: proxy: default: <backend_proxy_config> [ admin_api: <backend_proxy_config> ] [ compactor: <backend_proxy_config> ] [ distributor: <backend_proxy_config> ] [ ingester: <backend_proxy_config> ] [ querier: <backend_proxy_config> ] [ query_frontend: <backend_proxy_config> ]
To configure the gateway via command-line flags, make sure that each flag contains the path to the equivalent configuration field.
To configure the distributor backend proxy URL, use the flag
Likewise, given a configuration file field path of
gateway.proxy.query_frontend.url, the flag is
Note: A path is joined by a period (
.) and dashes (
-). Underscores (
_) are invalid within these command-line flags.
backend_proxy section specifies the URL and configuration of the backend that you want proxied.
A URL is only required for the
default proxy configuration. All other proxy configurations use the
default URL if it is not explicitly configured.
Other proxy configurations do not inherit any other configuration from the
default proxy configuration.
url: <url> | default = <gateway.proxy.default.url> [ enable_keepalive: <boolean> | default = true ] [ tls_enabled: <boolean> | default = false ] [ tls_ca_path: <string> | default = "" ] [ tls_server_name: <string> | default = "" ] [ tls_cert_path: <string> | default = "" ] [ tls_key_path: <string> | default = "" ] [ tls_insecure_skip_verify: <boolean> | default = false ] [ read_timeout: <duration> | default = "120s" ] [ write_timeout: <duration> | default = "30s" ]
Client-side load balancing
If you use a backend proxy URL that beings with
dns:///, the gateway creates a gRPC proxy with client-side round-robin load balancing instead of the default HTTP reverse proxy.
To configure client-side load balancing for requests to the distributors only, set the
Note: There are three
/characters in the preceding DNS URL, which means that you are using the default DNS authority. For details about DNS URLs, refer to RFC 4501.
Client-side load balancing is useful in ensuring that distributors are evenly loaded with requests.
Kubernetes Services are not load balancers; initial TCP connections are made using a random endpoint. After the connection is established, the same remote-write client talks to the same distributor server for its lifetime. This can mean an uneven load for your distributors and worse cluster performance overall.
The GET gateway solves this problem by exposing an HTTP server for receiving the client requests while using gRPC to talk to the distributors. The gRPC proxy maintains a list of endpoints that are returned from the DNS lookup, and keeps persistent connections to each endpoint. The proxies are also configured to perform per-request, client-side load balancing across the endpoints.
The Gateway exposes the ring endpoints for distributors, ingesters, and compactors. These
admin scope permissions.
/distributor/ringfor the distributors
/ingester/ringfor the ingesters
/compactor/ringfor the compactors
Related Enterprise Traces resources
Distributed tracing with Grafana: From Tempo OSS to Enterprise
Learn about our open source tracing database, Tempo, a scalable way to store and query distributed traces generated by any open source tracing protocol.
Get started with distributed tracing with Grafana Tempo and Grafana Enterprise Traces
In this ObservabilityCON session, find out the latest features in the open source tracing project and the new addition to Grafana Enterprise Stack.
Correlate your metrics, logs, and traces with Grafana Stack
Show how Grafana’s opinionated stack will allow you to correlate your metrics, logs, and traces, all under one roof.