Distribute Prometheus metrics scrape load
A good predictor for the size of an agent deployment is the number of Prometheus targets each agent scrapes. Clustering with target auto-distribution allows a fleet of agents to work together to dynamically distribute their scrape load, providing high-availability.
Note: Clustering is a beta feature. Beta features are subject to breaking changes and may be replaced with equivalent functionality that covers the same use case.
Before you begin
- Familiarize yourself with how to configure existing Grafana Agent installations.
- Configure Prometheus metrics collection.
- Configure clustering of agents.
- Ensure that all of your clustered agents have the same config file.
Steps
To distribute Prometheus metrics scrape load with clustering:
Add the following block to all
prometheus.scrape
components which should use auto-distribution:riverclustering { enabled = true }
Restart or reload agents for them to use the new configuration.
Validate that auto-distribution is functioning:
Using the UI on each agent, navigate to the details page for one of the
prometheus.scrape
components you modified.Compare the Debug Info sections between two different agents to ensure that they are not scraping the same sets of targets.