--- title: "Install Prometheus Operator with Grafana Cloud for Kubernetes | Grafana Cloud documentation" description: "Install Prometheus Operator with Grafana Cloud for Kubernetes You can install Prometheus Operator in a Kubernetes Cluster, configure it to scrape an endpoint, and send scraped metrics to Grafana Cloud." --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Install Prometheus Operator with Grafana Cloud for Kubernetes You can install Prometheus Operator in a Kubernetes Cluster, configure it to scrape an endpoint, and send scraped metrics to Grafana Cloud. [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and Kubernetes controller code. Together, these abstract away the management and implementation details of running a given service on Kubernetes. For more about Kubernetes Operators, refer to the [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). The Prometheus Operator installs a set of Kubernetes custom resources that simplify Prometheus deployment and configuration. For example, when you use the `ServiceMonitor` custom resource, you can configure how to monitor Kubernetes services in Kubernetes YAML manifests instead of Prometheus configuration code. The Operator controller then communicates with the Kubernetes API server to add service `/metrics` endpoints and automatically generate the required Prometheus scrape configurations for the configured services. To learn more, refer to [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator). ## Configuration steps The steps for configuration are: - Install Prometheus Operator into the Kubernetes Cluster. - Configure RBAC permissions for Prometheus. - Deploy Prometheus into the Cluster using the Operator. - Expose the Prometheus server as a service. - Create a ServiceMonitor. ## Before you begin Before you begin, have the following items available to you: - A Kubernetes `>=1.16.0` Cluster with [role-based access control](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/platform/rbac.md) (RBAC) enabled - A Grafana Cloud account - A Grafana Cloud access policy token with the `metrics:write` scope. To create a Grafana Cloud access policy, refer to [Create a Grafana Cloud Access Policy](/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/create-access-policies/). - The `kubectl` command-line tool installed on your local machine and configured to connect to your Cluster. Refer to [install `kubectl`](https://kubernetes.io/docs/tasks/tools/). ## Install Prometheus Operator Install Prometheus Operator into the Kubernetes Cluster. This includes all of Prometheus Operator’s Kubernetes custom resource definitions (CRDs) that define the Prometheus, Alertmanager, and ServiceMonitor abstractions used to configure the monitoring stack. In this step, you also deploy a Prometheus Operator controller into the Cluster. > Note > > With these instructions, you install everything into the `default` Namespace. To install Prometheus Operator into another Namespace, refer to the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator#quickstart). 1. Install the Operator using the `bundle.yaml` file in the Prometheus Operator GitHub repository: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml ``` You should see the following output: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created clusterrole.rbac.authorization.k8s.io/prometheus-operator created deployment.apps/prometheus-operator created serviceaccount/prometheus-operator created service/prometheus-operator created ``` `bundle.yaml` installs CRDs for Prometheus objects as well as a Prometheus Operator controller and service. > **Note:** With these instructions, you install everything into the `default` Namespace. To install Prometheus Operator into another Namespace, refer to the [Prometheus Operator documentation](https://github.com/prometheus-operator/prometheus-operator#quickstart). 2. Verify that the Prometheus Operator installation succeeded by using `kubectl`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl get deploy ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 3m21s ``` ## Configure Prometheus RBAC Permissions Configure RBAC privileges for the Prometheus server using a ClusterRole. Then bind this ClusterRole to a ServiceAccount using a ClusterRoleBinding object. Prometheus needs Kubernetes API access to discover targets and pull ConfigMaps. For more about permissions granted in this section, refer to [RBAC](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/platform/rbac.md). 1. Create a directory to store any Kubernetes manifests used in these instructions, and `cd` into it: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none mkdir operator_k8s cd operator_k8s ``` 2. Create a manifest file called `prom_rbac.yaml` using an editor, and paste in the following Kubernetes manifest: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none apiVersion: v1 kind: ServiceAccount metadata: name: prometheus --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/metrics - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: [""] resources: - configmaps verbs: ["get"] - apiGroups: - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: default ``` This creates a ServiceAccount called `prometheus` and binds it to the `prometheus` ClusterRole. The manifest grants the ClusterRole `get`, `list`, and `watch` Kubernetes API privileges. 3. Save and close the manifest when editing is complete. 4. Create the objects using `kubectl`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl apply -f ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created ``` Now that Prometheus has Kubernetes API access, you can deploy it into the Cluster. ## Deploy Prometheus In this step, you launch a 2-replica HA Prometheus deployment into your Kubernetes Cluster using a `Prometheus` resource defined by Prometheus Operator. This `Prometheus` resource encodes domain-specific Prometheus configuration into a set of readily configurable YAML fields. Instead of managing Prometheus configuration files and learn Prometheus configuration syntax, you can toggle many important configuration parameters by modifying `Prometheus` object variables in a Kubernetes manifest. 1. Create a file called `prometheus.yaml`, and paste in the following manifest: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus labels: app: prometheus spec: image: quay.io/prometheus/prometheus:v2.22.1 nodeSelector: kubernetes.io/os: linux replicas: 2 resources: requests: memory: 400Mi securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus version: v2.22.1 serviceMonitorSelector: {} ``` Notice that `kind` is set to `Prometheus` and not `Deployment` or `Pod`. The `Prometheus` resource abstracts away the underlying controllers and ConfigMaps into streamlined objects used to manipulate Prometheus infrastructure components. The manifest: - Names the `Prometheus` resource `prometheus`. - Gives it an `app: prometheus` label. - Sets the container image used to run Prometheus. - Restricts its deployment to Linux nodes. - Ensures each replica has `400Mi` of memory available to it. - Configures its Security Context. To learn more, refer to [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). - Sets the ServiceAccount it will use. - Sets the Prometheus version. - Instructs Prometheus to automatically pick up all configured `ServiceMonitor` resources using `{}`. You’ll create a `ServiceMonitor` to get Prometheus to scrape its own metrics. 2. Save and close the file. 3. Deploy the manifest into your Cluster using `kubectl apply -f`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl apply -f ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none prometheus.monitoring.coreos.com/prometheus created ``` > **Note:** You deployed Prometheus into the `default` Namespace. To deploy Prometheus in another Namespace, use the `-n namespace_name` flag with `kubectl` or set the `namespace` field for the resource in a Kubernetes manifest file. 4. Verify the deployment using `kubectl get`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl get prometheus ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none NAME VERSION REPLICAS AGE prometheus v2.22.1 2 32s ``` 5. Check the underlying Pods using `get pod`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl get pod ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none NAME READY STATUS RESTARTS AGE prometheus-operator-79cd654746-mdfp6 1/1 Running 0 33m prometheus-prometheus-0 2/2 Running 1 57s prometheus-prometheus-1 2/2 Running 1 57s ``` Prometheus is up and running in the Cluster. ## Create a Prometheus service Expose Prometheus using a service. To create the Prometheus service: 1. Open a manifest file called `prom_svc.yaml` and paste in the following definitions: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none apiVersion: v1 kind: Service metadata: name: prometheus labels: app: prometheus spec: ports: - name: web port: 9090 targetPort: web selector: app.kubernetes.io/name: prometheus sessionAffinity: ClientIP ``` This manifest configures: - Sets the service name to `prometheus`. - Creates an `app: prometheus` `Service` label. - Exposes port `9090` on a Cluster-wide stable IP address and forwards it to the two Prometheus Pods at their default `web` port (`9090`). - Selects the Prometheus Pods as targets using the `app: prometheus` label. - Uses `sessionAffinity: ClientIP` to ensure that connections from a particular client get forwarded to the same Pod. 2. Save and close the file. 3. Create the service using `kubectl apply -f`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl apply -f prom_svc.yaml ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none service/prometheus created ``` 4. Check your work using `kubectl get`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl get service ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.245.0.1 443/TCP 27h prometheus ClusterIP 10.245.106.105 9090/TCP 26h prometheus-operated ClusterIP None 9090/TCP 8m52s prometheus-operator ClusterIP None 8080/TCP 41m ``` The `prometheus` service is up and running. 5. To access the Prometheus server, forward a local port to the Prometheus service running inside of the Kubernetes Cluster: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl port-forward svc/prometheus 9090 ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090 ``` 6. Navigate to `http://localhost:9090` to access the Prometheus interface: 7. Click **Status**, then **Targets** to see any configured scrape targets. This should be empty, as nothing is configured to scrape. ## Create a Prometheus ServiceMonitor Create a ServiceMonitor so that the Prometheus server scrapes its own metrics endpoint. A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource. Instead of modifying a Prometheus configuration file, update a ConfigMap object and deploy the new configuration so that Prometheus Operator automatically hooks in new ServiceMonitors to your running Prometheus deployment. To create the ServiceMonitor: 1. Create a file called `prometheus_servicemonitor.yaml`, and paste in the following ServiceMonitor resource definition: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: prometheus-self labels: app: prometheus spec: endpoints: - interval: 30s port: web selector: matchLabels: app: prometheus ``` The resource definition configures the following: - Names the ServiceMonitor `prometheus-self`. - Gives it an `app: prometheus` label. - Sets the scrape interval to `30s`, and scrapes the `web` port (defined in the `prometheus` `Service`). - Selects the `prometheus` `Service` to scrape using the `matchLabels` selector with `app: prometheus`. 2. Save and close the file. 3. Deploy the using `kubectl apply -f`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl apply -f prometheus_servicemonitor.yaml ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none servicemonitor.monitoring.coreos.com/prometheus-self created ``` Prometheus Operator should update your Prometheus configuration immediately. In some cases, you may need to wait a minute or so for changes to propagate. ## Verify your updates Forward a port to your Prometheus server and check its configuration for verification. **And replace with this one?** 1. Find the name of the Prometheus service. This will be in the format <your\_release\_name>-kube-prometheus-stack-prometheus, but could be truncated to fit naming length restrictions. ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl --namespace default get service ``` 2. Use port-forward with that Prometheus service by running the following command: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl --namespace default port-forward svc/ 9090 ``` 3. Navigate to **Status**, and then **Targets** in the Prometheus interface: You should see the two Prometheus replicas as scrape targets. 4. Navigate to **Graph** to test metrics collection: 5. In the **Expression** box, type `prometheus_http_requests_total`, and press `ENTER`. You should see a list of scraped metrics and their values. These are HTTP request counts for various Prometheus server endpoints. If you’re having trouble configuring a ServiceMonitor, refer to [Troubleshooting ServiceMonitor Changes](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/platform/troubleshooting.md#troubleshooting-servicemonitor-changes). Now you have configured Prometheus to scrape itself and store metrics locally. ## Create a Kubernetes Secret to store Grafana Cloud credentials Configure Prometheus to send the scraped metrics to Grafana Cloud. Before configuring Prometheus’s `remote_write` feature to send metrics to Grafana Cloud, you must create a Kubernetes Secret to store your Grafana Cloud Metrics username and password. To create the Secret: 1. Navigate to your stack in the Cloud Portal, and click **Details** next to the Prometheus panel to find your username. Your password corresponds to a Cloud Access Policy token that you can generate by clicking on **Generate now** in this same panel. To create a Cloud Access Policy, refer to [Create a Grafana Cloud Access Policy](/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/access-policies/create-access-policies/). After noting your Cloud Prometheus username and password, you can create the Kubernetes Secret using a manifest file or directly using `kubectl`. With these instructions, you create it directly using `kubectl`. To learn more about Kubernetes Secrets, refer to [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/). 2. Run the following command to create a Secret called `kubepromsecret`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl create secret generic kubepromsecret \ --from-literal=username=\ --from-literal=password='' ``` > **Note:** If you deployed your monitoring stack in a namespace other than `default`, append the `-n` flag with the appropriate namespace to the above command. For more about this command, refer to [Managing Secrets using kubectl](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/) from the official Kubernetes documentation. ## Configure Prometheus remote\_write and metrics deduplication Configure `remote_write` to send Cluster metrics to Grafana Cloud and to deduplicate metrics. The `remote_write` Prometheus feature allows you to send metrics to remote endpoints for long-term storage and aggregation. Grafana Cloud’s deduplication feature allows you to deduplicate metrics sent from high-availability Prometheus pairs, which reduces your active series usage. You enable `remote_write` by modifying the `prometheus` resource you created earlier. To configure Prometheus remote\_write and metrics deduplication: 1. Open the `prometheus.yaml` manifest using your editor: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus labels: app: prometheus spec: image: quay.io/prometheus/prometheus:v2.22.1 nodeSelector: kubernetes.io/os: linux replicas: 2 resources: requests: memory: 400Mi securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus version: v2.22.1 serviceMonitorSelector: {} ``` 2. Add the following section to the end of the resource definition: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none . . . remoteWrite: - url: "" basicAuth: username: name: kubepromsecret key: username password: name: kubepromsecret key: password replicaExternalLabelName: "__replica__" externalLabels: cluster: "" ``` The resource definition configures the following: - Sets the `remote_write` URL corresponding to Grafana Cloud’s Prometheus metrics endpoint. To find the `/api/prom/push` URL, username, and password for your metrics endpoint, click **Details** in the Prometheus card of the [Cloud Portal](/docs/grafana-cloud/account-management/cloud-portal/). - Configures a `basicAuth` username and password that references the Secret created in the earlier step named `kubepromsecret` - Selects the `username` and `password` keys of this Secret - Configures Grafana Cloud metrics deduplication using the `replicaExternalLabelName` and `externalLabels` fields - Sets `cluster` to a value that identifies your Prometheus HA Cluster. To learn more, refer to [Deduplicating metrics data sent from high-availability Prometheus pairs](/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/client-side-filtering/). 3. Save and close the file. 4. Apply the changes using `kubectl apply -f`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl apply -f prometheus.yaml ``` ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none prometheus.monitoring.coreos.com/prometheus configured ``` At this point, you’ve successfully configured your Prometheus instances to `remote_write` scraped metrics to Grafana Cloud. 5. Verify that your changes have propagated to your running Prometheus instances using `port-forward`: ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```none kubectl port-forward svc/prometheus 9090 ``` 6. Navigate to `http://localhost:9090` in your browser, and then **Status** and **Configuration**. Verify that the `remote_write` and `external_labels` blocks you appended earlier have propagated to your running Prometheus instances. It may take a minute or two for Prometheus Operator to pick up the new configuration. In the final step, you’ll query your Cluster metrics in Grafana Cloud. ## Access your Prometheus metrics in Grafana Cloud Now that your Prometheus instances send their scraped metrics to Grafana Cloud, you can query and visualize these metrics from the Grafana Cloud platform. 1. From the Cloud Portal, click **Log In** next to the Grafana card to log in to Grafana. 2. Click **Explore** in the left-side menu. 3. In the **PromQL** query box, enter the same metric you tested earlier, `prometheus_http_requests_total`, and press `SHIFT + ENTER`. You should see a graph of time-series data corresponding to different labels of the `prometheus_http_requests_total` metric. Grafana queries this data from the Grafana Cloud Metrics data store, *not* your local Cluster. 4. Navigate to [Kubernetes Monitoring](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/navigate-k8s-monitoring/#navigate-to-kubernetes-monitoring), and click **Configuration** on the main menu. 5. Click the **Metrics status** tab to view the data status. Your data begins populating in the view as the system components begin scraping and sending data to Grafana Cloud. [**Metrics status** tab with status indicators for one Cluster](/media/docs/grafana-cloud/k8s/metrics-status-9-18.png) ## Next steps Now that you have installed and configured Prometheus Operator, you can: - Create dashboards and panels to visualize and alert on this data. To learn more, refer to [Dashboard overview](/docs/grafana/latest/dashboards/) and [Panel overview](/docs/grafana/latest/panels-visualizations/). - Add additional endpoints to scrape, such as the Kubernetes API or `kubelet` metrics from the Kubernetes nodes. To see a fully configured Prometheus Kubernetes stack in action, refer to [kube-prometheus](https://kubernetes.io/docs/reference/access-authn-authz/rbac/).