MetricsPrometheus

Metrics — Prometheus

To get Prometheus metrics into Grafana Cloud, configure Prometheus to push scraped samples using remote_write. remote_write allows you to forward scraped samples to compatible remote storage endpoints. To learn more, please see remote_write from the Prometheus docs.

Sending data from Prometheus

To send Prometheus data to Grafana Cloud Metrics you will need a running Prometheus instance in your environment, as in the following diagram:

Local Prometheus Architecture

Add the following remote_write snippet to your Prometheus configuration file (often prometheus.yml). You can find the /api/prom/push URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal:

remote_write:
- url: <Your Metrics instance remote_write endpoint>
  basic_auth:
    username: <Your Metrics instance ID>
    password: <Your Grafana.com API Key>

Sending data from multiple Prometheus instances

When sending metrics from multiple Prometheus instances, you can use the external_labels parameter to label time series data with an instance identifier. Append any external_labels to the global section of your Prometheus configuration file. You can find the /api/prom/push URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal:

First Prometheus instance:

global:
  external_labels:
    origin_prometheus: prometheus01
remote_write:
- url: <Your Metrics instance remote_write endpoint>
  basic_auth:
    username: <Your Metrics instance ID>
    password: <Your Grafana.com API Key>

Second Prometheus instance:

global:
  external_labels:
    origin_prometheus: prometheus02
remote_write:
- url: <Your Metrics instance remote_write endpoint>
  basic_auth:
    username: <Your Metrics instance ID>
    password: <Your Grafana.com API Key>

Sending data from multiple high-availability Prometheus instances

If you’re shipping data to Grafana Cloud Metrics from multiple Prometheus instances in a high-availability (HA) configuration, you should use the cluster and __replica__ labels to deduplicate data received at the Grafana Cloud Metrics endpoint. Add these to the external_labels section in your Prometheus configuration file. You can find the /api/prom/push URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal:

First HA Prometheus instance:

global:
  external_labels:
    cluster: prom-team1
    __replica__: replica1
remote_write:
- url: <Your Metrics instance remote_write endpoint>
  basic_auth:
    username: <Your Metrics instance ID>
    password: <Your Grafana.com API Key>

Second HA Prometheus instance:

global:
  external_labels:
    cluster: prom-team1
    __replica__: replica2
remote_write:
- url: <Your Metrics instance remote_write endpoint>
  basic_auth:
    username: <Your Metrics instance ID>
    password: <Your Grafana.com API Key>

The cluster label will identify the HA Prometheus cluster shipping data to Grafana Cloud Metrics. The __replica__ label should be set to a unique value within the HA Prometheus cluster. It will be dropped when Grafana Cloud Metrics deduplicates and ingests the data.

To learn more about Prometheus configuration parameters, please see the Prometheus.io Configuration Docs.

The Monitoring a Linux host using Prometheus and node_exporter provides a complete start to finish example of installing a local Prometheus instance and using remote_write to send metrics from the instance to Grafana Cloud.