Documentation Grafana Cloud Monitor infrastructure Kubernetes Monitoring Configure Kubernetes Monitoring Configure with other methods remote_write Prometheus ConfigMap

Configure remote_write with a Prometheus ConfigMap

If you have Prometheus installed and running in your cluster, and you configured using a Kubernetes ConfigMap, you can send scraped samples to Grafana Cloud using Prometheus’s remote_write feature.

Before you begin

Before you begin, you must have the following items:

• A Kubernetes >=1.16.0 cluster.
• A Grafana Cloud Standard account. To create an account, refer to Get started.
• A Grafana Cloud access policy token with the metrics:write scope. To create a Grafana Cloud access policy, refer to Create a Grafana Cloud Access Policy.
• The kubectl command-line tool installed on your local machine and configured to connect to your cluster. To install, refer to install kubectl.
• The Prometheus monitoring system installed and running in your cluster as a Deployment, configured using a ConfigMap

To install Prometheus Operator in your cluster, refer to Install Prometheus Operator with Grafana Cloud for Kubernetes. Prometheus Operator abstracts away much of Prometheus’s configuration and management overhead.

Modify Prometheus ConfigMap

1. To locate your Grafana Cloud Metrics username and password, navigate to your stack in the Cloud Portal and click Details next to the Prometheus panel.

   Your password corresponds to a Cloud Access Policy token that you can generate by clicking on Generate now in this same panel. To create a Cloud Access Policy, refer to Create a Grafana Cloud Access Policy.

2. Note your username and password.

3. To inject your username and password into your Prometheus configuration file, modify the Kubernetes ConfigMap resource which contains Prometheus’s configuration.

   Unfortunately Prometheus does not support pulling environment variables from the execution environment. Therefore, you cannot readily use a Kubernetes Secret object in this case. Using Secrets or envsubst goes beyond the scope of this content.

4. Locate the ConfigMap manifest and open it in an editor. It will look similar to the following example. This ConfigMap is installed in the monitoring Namespace, where the Prometheus Deployment should also be running.

   Copy

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: prometheus
     namespace: monitoring
   data:
     prometheus.yml: |-
       global:
         scrape_interval: 5s
         evaluation_interval: 5s
       rule_files:
         - /etc/prometheus/prometheus.rules
       alerting:
         alertmanagers:
         - scheme: http
           static_configs:
           - targets:
             - "alertmanager.monitoring.svc:9093"
   
   . . .

5. Modify this ConfigMap by adding the following remote_write configuration block:

   Copy

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: prometheus_v2
     namespace: monitoring
   data:
     prometheus.yml: |-
       global:
         scrape_interval: 5s
         evaluation_interval: 5s
   
   . . .
   
   remote_write:
   - url: <Your Metrics instance remote_write endpoint>
     basic_auth:
       username: <your_grafana_cloud_metrics_username>
       password: <your_grafana_cloud_metrics_password>

   To find the /api/prom/push URL, username, and password for your metrics endpoint, click Details in the Prometheus card of the Cloud Portal.

   This block creates a default remote_write configuration that sends samples to the Cloud Metrics Prometheus endpoint. It also sets the authorization header on remote_write requests with your Grafana Cloud credentials. To tune the default remote_write parameters, refer to Remote Write Tuning.

   Make sure you give the ConfigMap a new versioned name as well. To do so, append a suffix like _v2.

6. Save and close the file.

Update the running Prometheus Deployment

1. To make your configuration changes in the cluster, update the Prometheus Deployment with the newly versioned ConfigMap.

   Note

   You can make configuration changes in Kubernetes clusters in many different ways. These steps focus on configuring remote_write, and are not meant to cover blue-green or production Prometheus rollout scenarios.

   In the previous step, you assigned the ConfigMap a new name. When you update the ConfigMap reference in the Deployment, Kubernetes redeploys any Pods using the old ConfigMap.

2. Open the Prometheus Deployment in an editor. It will look similar to the following example. This file may vary depending on how you configured and deployed Prometheus. The above manifest defines a 2-Pod Prometheus Deployment that references a ConfigMap called prometheus. The prometheus.yml key containing Prometheus’s configuration is mounted to /etc/prometheus/prometheus.yml.

   Copy

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: prometheus-deployment
     labels:
       app: prometheus
   spec:
     replicas: 2
     selector:
   
   . . .
   
     template:
       metadata:
         labels:
           app: prometheus
   
   . . .
   
       spec:
         containers:
         - name: prometheus
           image: prometheus
           ports:
           - containerPort: 9090
           volumeMounts:
             - name: config-volume
               mountPath: /etc/prometheus/
         volumes:
           - name: config-volume
             configMap:
               name: prometheus

3. To update this Deployment, change the ConfigMap:

   Copy

   . . .
         volumes:
           - name: config-volume
             configMap:
               name: prometheus_v2

   You update the configMap’s name field from prometheus to prometheus_v2 to reference the new ConfigMap defined earlier.

4. When you’re done, save and close the file.

5. Roll out the changes using kubectl apply -f:

   Copy

   kubectl apply -f <your_prometheus_deployment_manifest>.yaml

Check your work

To verify that your running Prometheus instance is remote_writing correctly:

1. Get the Prometheus server’s service name:

   Copy

   kubectl get svc

2. Use port-forward to forward a local port to the Prometheus service:

   Copy

   kubectl --namespace monitoring port-forward svc/<prometheus-service-name> 9090:80

   Replace monitoring with the appropriate namespace, and <prometheus-service-name> with the name of the Prometheus service.

3. Navigate to http://localhost:9090 in your browser, and then Status and Configuration.

4. Verify that the remote_write block you appended above has propagated to your running Prometheus instance configuration.

5. Log in to your Grafana instance to begin querying your cluster data.

6. Navigate to Kubernetes Monitoring, and click Configuration on the main menu.

7. Click the Metrics status tab to view the data status. Your data begins populating in the view as the system components begin scraping and sending data to Grafana Cloud.