MetricsPrometheusConfiguring Prometheus remote_write for Kubernetes deploymentsConfiguring remote_write with Prometheus Operator

Configuring remote_write with Prometheus Operator

This guide assumes you have either Prometheus Operator or kube-prometheus installed and running in your Kubernetes cluster. Prometheus Operator is a sub-component of the kube-prometheus stack. To learn how to install Prometheus Operator, please see the Prometheus Operator GitHub repository and to learn how to install kube-prometheus, please see kube-prometheus.

Prometheus Operator implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and controller code that abstract away the management and implementation details of running a given service on Kubernetes. To learn more about Kubernetes Operators, please see Operator pattern from the Kubernetes docs.

The Prometheus Operator provides a set of Kubernetes Custom Resources that simplify Prometheus, Grafana and Alertmanagemer deployment and configuration. For example, using the ServiceMonitor Custom Resource, you can configure how groups of Kubernetes services should be monitored in YAML manifests. The Operator controller will then communicate with the K8s API server to monitor Service endpoints and automatically generate the required Prometheus scrape configurations for the configured Services. To learn more about Prometheus Operator, please see the Prometheus Operator GitHub repository.

kube-prometheus configures Prometheus Operator with a default Prometheus-Alertmanager-Grafana stack and sets up preconfigured Grafana dashboards and Alertmanager alerts. It also configures a set of Prometheus scrape targets and sets up node-exporter and kube-state-metrics.

If you used the Helm package manager to install either of these components, please see the relevant guide.

Step 1 — Create a Kubernetes Secret to store Grafana Cloud credentials

Begin by creating a Kubernetes Secret to store your Grafana Cloud Metrics username and password.

You can find your username by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.

Your password corresponds to an API key that you can generate by clicking on Generate now in this same panel. To learn how to create a Grafana Cloud API key, please see Create a Grafana Cloud API key

Once you’ve noted your Cloud Prometheus username and password, create the Kubernetes secret. You can create a Secret by using a manifest file or create it directly using kubectl. In this guide we’ll create it directly using kubectl. To learn more about Kubernetes Secrets, please consult Secrets from the Kubernetes docs.

Run the following command to create a Secret called kubepromsecret:

kubectl create secret generic kubepromsecret \
  --from-literal=username=<your_grafana_cloud_prometheus_username>\
  --from-literal=password='<your_grafana_cloud_API_key>'\
  -n monitoring

If you deployed your monitoring stack in a namespace other than monitoring, change the -n monitoring flag to the appropriate namespace in the above command. To learn more about this command, please see Managing Secret using kubectl from the official Kubernetes docs.

Now that you’ve created a Secret to store your Grafana Cloud credentials, you can move on to modifying your Prometheus configuration.

Step 2 — Modify the Prometheus manifest configuration

Begin by locating the manifest file for the Prometheus custom resource running in your cluster. You can use a Prometheus custom resource to define and control one or more Prometheus replicas in your Kubernetes cluster.

If you’re using kube-prometheus and deployed its default stack, this will be prometheus-prometheus.yaml in the manifests directory of the kube-prometheus GitHub repo.

If you’re using Prometheus Operator, you have to first define and deploy one or more Prometheus instances using the Prometheus Custom Resource Definition (CRD) created by Prometheus Operator. You can use the Prometheus manifest from kube-prometheus to help you begin. Installing and configuring Prometheus in your cluster goes beyond the scope of this guide.

Once you’ve located the file, open it in your favorite text editor. You should see something like the following:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.22.1
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.22.1

Note the manifest file may vary depending on your specific Prometheus deployment.

Append the following remote_write configuration block after the version parameter:

. . .
  version: v2.22.1
  remoteWrite:
  - url: "https://prometheus-us-central1.grafana.net/api/prom/push"
    basicAuth:
      username:
        name: kubepromsecret
        key: username
      password:
        name: kubepromsecret
        key: password

Here we set the remote_write URL corresponding to Grafana Cloud’s Prometheus metrics endpoint. We also configure a basicAuth username and password referencing the Secret created in the previous step named kubepromsecret. We select the username and password keys of this Secret.

Save and close the file when you’re done editing.

Roll out the changes using kubectl apply -f:

kubectl apply -f prometheus-prometheus.yaml -n monitoring

Replace prometheus-prometheus.yaml with the appropriate filename and monitoring with the namespace into which the Prometheus stack has been installed.

You should see the following output:

prometheus.monitoring.coreos.com/k8s configured

At this point, you’ve successfully configured your Prometheus instances to remote_write scraped metrics to Grafana Cloud. You can verify that your changes have propagated to your running Prometheus instances using port-forward:

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Replace namespace with the appropriate namespace, and prometheus-k8s with the Prometheus Service name.

Navigate to http://localhost:9090 in your browser, and then Status and Configuration. Verify that the remote_write block you appended above has propagated to your running Prometheus instances.

Finally, log in to your Grafana instance to begin querying your cluster data. You can use the Billing/Usage dashboard to inspect incoming data rates in the last 5 minutes to confirm the flow of data to Grafana Cloud.