Configure remote_write with a Prometheus ConfigMap
In this guide, you’ll learn how to configure Prometheus to ship scraped samples to Grafana Cloud using Prometheus’s
Follow this guide if you have Prometheus installed and running in your cluster, and configured using a Kubernetes ConfigMap.
Before you begin
Before you begin, you should have the following items available to you:
- A Kubernetes
- A Grafana Cloud Standard account. To learn how to create an account, see Grafana Cloud Quickstarts.
- A Grafana Cloud API key with the
MetricsPublisherrole. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key.
kubectlcommand-line tool installed on your local machine and configured to connect to your cluster. See how to install
- The Prometheus monitoring system installed and running in your cluster as a deployment, configured using a ConfigMap. Installing Prometheus goes beyond the scope of this guide. To learn how to install Prometheus Operator in your cluster, see Install Prometheus Operator with Grafana Cloud for Kubernetes. Prometheus Operator abstracts away much of Prometheus’s configuration and management overhead.
Modify Prometheus ConfigMap
Locate your Grafana Cloud Metrics username and password by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.
Your password corresponds to an API key that you can generate by clicking on Generate now in this same panel. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key.
Once you’ve noted your username and password, inject it into your Prometheus configuration file by modifying the Kubernetes ConfigMap resource containing Prometheus’s configuration.
Unfortunately Prometheus does not support pulling environment variables from the execution environment so you can’t readily use a Kubernetes Secret object in this case. Using Secrets or
envsubstgoes beyond the scope of this guide.
Locate the ConfigMap manifest and open it in an editor. It will look similar to the following example:
apiVersion: v1 kind: ConfigMap metadata: name: prometheus namespace: monitoring data: prometheus.yml: |- global: scrape_interval: 5s evaluation_interval: 5s rule_files: - /etc/prometheus/prometheus.rules alerting: alertmanagers: - scheme: http static_configs: - targets: - "alertmanager.monitoring.svc:9093" . . .
This ConfigMap is installed in the
monitoringNamespace, where the Prometheus deployment should also be running.
Modify this ConfigMap by adding the following
apiVersion: v1 kind: ConfigMap metadata: name: prometheus_v2 namespace: monitoring data: prometheus.yml: |- global: scrape_interval: 5s evaluation_interval: 5s . . . remote_write: - url: <Your Metrics instance remote_write endpoint> basic_auth: username: <your_grafana_cloud_metrics_username> password: <your_grafana_cloud_metrics_password>
You can find the
/api/prom/pushURL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal.
This block creates a default
remote_writeconfiguration that ships samples to the Cloud Metrics Prometheus endpoint. It also sets the authorization header on
remote_writerequests with your Grafana Cloud credentials. To tune the default
remote_writeparameters, see Remote Write Tuning from the Prometheus documentation.
Be sure to give the ConfigMap a new versioned name as well, by appending a suffix like
When you’re done, save and close the file.
In the next step you’ll roll out this updated configuration into your cluster.
Update the running Prometheus deployment
To roll out your configuration changes, update the Prometheus deployment with the new versioned ConfigMap.
Note: You can roll out configuration changes in Kubernetes clusters in many different ways. The steps in this guide focus on configuring
remote_writeand are not meant to cover blue-green or production Prometheus rollout scenarios.
Given that in the previous step you assigned the ConfigMap a new name, when you update the ConfigMap reference in the deployment, Kubernetes redeploys any Pods using the old ConfigMap.
Open the Prometheus deployment in an editor. It will look similar to the following example:
apiVersion: apps/v1 kind: Deployment metadata: name: prometheus-deployment labels: app: prometheus spec: replicas: 2 selector: . . . template: metadata: labels: app: prometheus . . . spec: containers: - name: prometheus image: prometheus ports: - containerPort: 9090 volumeMounts: - name: config-volume mountPath: /etc/prometheus/ volumes: - name: config-volume configMap: name: prometheus
This file may vary depending on how you configured and deployed Prometheus. The above manifest defines a 2-pod Prometheus deployment that references a ConfigMap called
prometheus.ymlkey containing Prometheus’s configuration is mounted to
To update this deployment, change the ConfigMap:
. . . volumes: - name: config-volume configMap: name: prometheus_v2
You update the
prometheus_v2to reference the new ConfigMap defined earlier.
When you’re done, save and close the file.
Roll out the changes using
kubectl apply -f:
kubectl apply -f <your_prometheus_deployment_manifest>.yaml
Check your work
At this point, you’ve configured Prometheus to
remote_write scraped metrics to Grafana Cloud. You can verify that your running Prometheus instance is remote_writing correctly using
To verify your changes:
Get the Prometheus server’s service name:
kubectl get svc
port-forwardto forward a local port to the Prometheus service:
kubectl --namespace monitoring port-forward svc/<prometheus-service-name> 9090:80
monitoringwith the appropriate namespace, and
<prometheus-service-name>with the name of the Prometheus service.
http://localhost:9090in your browser, and then Status and Configuration.
Verify that the
remote_writeblock you appended above has propagated to your running Prometheus instance configuration.
Finally, log in to your Grafana instance to begin querying your cluster data. You can use the Billing/Usage dashboard to inspect incoming data rates in the last 5 minutes to confirm the flow of data to Grafana Cloud.