Reducing Prometheus metrics usage
Once you’ve gathered a list of high-cardinality metrics, you can go through active series and prune metrics and labels that you don’t want to store in Grafana Cloud. You can do this in two different ways:
- Keeping a set of metrics that you explicitly define, and dropping everything else.
- Dropping a set of metrics that you explicitly define, and keeping everything else.
Both these methods use Prometheus’s
relabel_config parameter. To learn more about this parameter, please see
relabel_config from the Prometheus docs. The Advanced Service Discovery in Prometheus 0.14.0 blog post also contains many practical examples that may be helpful.
You can use Prometheus’s
metric_relabel_configs to drop some series before sending them to the remote cloud storage.
If you want to store these metrics locally but not send them to the remote cloud storage,
then you should use the
write_relabel_configs in the
remote_write section of your Prometheus config. To learn more about
remote_write, please see
remote_write from the official Prometheus docs.
If you don’t want to store those metrics locally, you can use the
metric_relabel_configs in the
scrape_config sections. To learn more, please see