Configure remote_write with Prometheus Operator

In this guide, you’ll learn how to configure Prometheus Operator to scrape an endpoint, and ship scraped metrics to Grafana Cloud.

Prometheus Operator implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and controller code that abstract away the management and implementation details of running a given service on Kubernetes. To learn more about Kubernetes Operators, see Operator pattern from the Kubernetes docs.

The Prometheus Operator provides a set of Kubernetes Custom Resources that simplify Prometheus, Grafana and Alertmanager deployment and configuration. For example, using the ServiceMonitor custom resource, you can configure how to monitor groups of Kubernetes services in YAML manifests. The Operator controller then communicates with the Kubernetes API server to monitor service endpoints and automatically generate the required Prometheus scrape configurations for the configured services. To learn more about Prometheus Operator, refer to the Prometheus Operator GitHub repository.

kube-prometheus configures Prometheus Operator with a default Prometheus-Alertmanager-Grafana stack and sets up preconfigured Grafana dashboards and Alertmanager alerts. It also configures a set of Prometheus scrape targets and sets up node-exporter and kube-state-metrics.

Before you begin

This guide assumes you have either Prometheus Operator or kube-prometheus installed and running in your Kubernetes cluster. Prometheus Operator is a sub-component of the kube-prometheus stack.

To learn how to install Prometheus Operator, see the Prometheus Operator GitHub repository.
To learn how to install kube-prometheus, see kube-prometheus.

If you used the Helm package manager to install either of these components, see the relevant guide:

Create a Kubernetes Secret to store Grafana Cloud credentials

To begin, you’ll create a Kubernetes Secret to store your Grafana Cloud Metrics username and password.

To create your Kubernetes Secret:

Find your username by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.
Your password corresponds to an API key that you can generate by clicking Generate now in the same panel. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key
Once you’ve noted your Cloud Prometheus username and password, run the following command to create a Secret, in this example, kubepromsecret:
```
kubectl create secret generic kubepromsecret \
  --from-literal=username=<your_grafana_cloud_prometheus_username>\
 --from-literal=password='<your_grafana_cloud_API_key>'\
 -n default
```
Note: If you deployed your monitoring stack in a namespace other than default, change the -n monitoring flag to the appropriate namespace in the above command. To learn more about this command, see Managing Secrets using kubectl from the official Kubernetes docs.
Instead of creating the secret directly as in this example, alternatively you can use a manifest file. To learn more about Kubernetes Secrets, refer to Secrets from the Kubernetes docs.

Now that you’ve created a Secret to store your Grafana Cloud credentials, you can move on to modifying your Prometheus configuration.

Modify the Prometheus manifest configuration

Locate the manifest file for the Prometheus custom resource running in your cluster.
- If you’re using kube-prometheus and deployed its default stack, this will be prometheus-prometheus.yaml in the manifests directory of the kube-prometheus GitHub repo.
- If you’re using Prometheus Operator, you have to first define and deploy one or more Prometheus instances using the Prometheus Custom Resource Definition (CRD) created by Prometheus Operator. You can use the Prometheus manifest from kube-prometheus to help you begin. Installing and configuring Prometheus in your cluster is beyond the scope of this guide.

Once you’ve located the file, open it in an editor. You should see something like the following:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.22.1
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.22.1

The manifest file may vary depending on your specific Prometheus deployment.

Edit and append the following remote_write configuration block after the version parameter:
```
  version: v2.22.1
  remoteWrite:
  - url: "<Your Metrics instance remote_write endpoint>"
    basicAuth:
      username:
        name: kubepromsecret
        key: username
      password:
        name: kubepromsecret
        key: password
```
Make the following edits:
- Set the remote_write URL to Grafana Cloud’s Prometheus metrics endpoint. You can find the /api/prom/push URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal.
- Update the basicAuth username and password fields based on the Secret you created in the previous step (named kubepromsecret in the example). Enter the username and password keys for the Secret.
Save and close the file when you’re done editing.
Roll out the changes using the following command:
```
kubectl apply -f prometheus-prometheus.yaml -n monitoring
```
Replace prometheus-prometheus.yaml with the appropriate filename and monitoring with the namespace into which the Prometheus stack has been installed.
You should see the following output:
```
prometheus.monitoring.coreos.com/k8s configured
```

Verify your updates

At this point, you’ve successfully configured your Prometheus instances to remote_write scraped metrics to Grafana Cloud. You can verify that your changes have propagated to your running Prometheus instances using port-forward.

To verify your changes:

Use port-forward by running this command:
```
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
```
Replace namespace with the appropriate namespace, and prometheus-k8s with the Prometheus service name.
Navigate to http://localhost:9090 in your browser, and then Status and Configuration.
Verify that the remote_write block you appended above has propagated to your running Prometheus instances.
Finally, log in to your Grafana instance to begin querying your cluster data.
You can use the Billing/Usage dashboard to inspect incoming data rates in the last 5 minutes to confirm the flow of data to Grafana Cloud.

Configure remote_write with Prometheus Operator

Before you begin

Create a Kubernetes Secret to store Grafana Cloud credentials

Modify the Prometheus manifest configuration

Verify your updates

Related documentation

Related resources from Grafana Labs