Mirror requests to a second Grafana Mimir cluster
Use request mirroring when you want to set up a Grafana Mimir testing cluster that receives the same series that are ingested by a primary cluster and when you don’t have control over the Prometheus remote write configuration.
If you have control over the Prometheus remote write configuration, we recommend that you configure two remote write entries in Prometheus. For more information about Prometheus remote write configuration, refer to Prometheus remote write reference.
Mirroring with Envoy proxy
You can use Envoy proxy to mirror HTTP requests to a secondary upstream cluster. From a network path perspective, run Envoy in front of both clusters’ distributors.
This approach enables the Envoy proxy to make requests to the primary Grafana Mimir cluster and then in the background, Envoy mirrors the requests to the secondary cluster.
The performance and availability of the secondary cluster have no impact on the requests to the primary cluster. The response to the client is always sent from the primary cluster.
Requests from Envoy to the secondary cluster are “fire and forget”, which means that Envoy doesn’t wait until the requests are completed to the secondary cluster before sending back the response to the client.
The following diagram illustrates a simplified network structure.
Example Envoy configuration
The following Envoy configuration shows an example that includes two Grafana Mimir clusters. Envoy listens on port 9900
and proxies all requests to mimir-primary:8080
, while also mirroring them to mimir-secondary:8080
.