Menu
Grafana Cloud Grafana Kubernetes Monitoring Other configuration methods Use remote_write to ship Prometheus metrics Install Prometheus Operator with Grafana Cloud for Kubernetes
Grafana Cloud

Install Prometheus Operator with Grafana Cloud for Kubernetes

In this guide, you’ll learn how to install Prometheus Operator in a Kubernetes (K8s) cluster, configure it to scrape an endpoint, and ship scraped metrics to Grafana Cloud.

Prometheus Operator implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and Kubernetes controller code. Together, these abstract away the management and implementation details of running a given service on Kubernetes. To learn more about Kubernetes Operators, see the Operator pattern from the Kubernetes docs.

The Prometheus Operator installs a set of Kubernetes custom resources that simplify Prometheus deployment and configuration. For example, using the ServiceMonitor custom resource, you can configure how to monitor Kubernetes services in K8s YAML manifests instead of Prometheus configuration code. The Operator controller then communicates with the K8s API server to add service /metrics endpoints and automatically generate the required Prometheus scrape configurations for the configured services. To learn more about Prometheus Operator, see the Prometheus Operator GitHub repository.

In this guide, you’ll begin by installing Prometheus Operator into the Kubernetes cluster. Next, you’ll launch a 2-replica high-availability (HA) Prometheus deployment into the cluster using the Operator, and then expose the Prometheus server as a service. Finally, you’ll create a ServiceMonitor to instruct Prometheus to scrape itself, and then ship those scraped metrics to Grafana Cloud.

Before you begin

Before you begin, you should have the following items available to you:

  • A Kubernetes >=1.16.0 cluster with role-based access control (RBAC) enabled.
  • A Grafana Cloud account.
  • A Grafana Cloud API key with the MetricsPublisher role. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key.
  • The kubectl command-line tool installed on your local machine and configured to connect to your cluster. See how to install kubectl.

Install Prometheus Operator

You’ll start by installing Prometheus Operator into the Kubernetes cluster. You’ll install all of Prometheus Operator’s Kubernetes custom resource definitions (CRDs) that define the Prometheus, Alertmanager, and ServiceMonitor abstractions used to configure the monitoring stack. You’ll also deploy a Prometheus Operator controller into the cluster.

Note: Following this guide, you’ll install everything into the default Namespace. To learn how to install Prometheus Operator into another Namespace, see the Prometheus Operator documentation.

  1. Install the Operator using the bundle.yaml file in the Prometheus Operator GitHub repository:

    kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

    You should see the following output:

    customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
serviceaccount/prometheus-operator created
service/prometheus-operator created

    bundle.yaml installs CRDs for Prometheus objects as well as a Prometheus Operator controller and service.

    Note: Following this guide, you’ll install everything into the default Namespace. To learn how to install Prometheus Operator into another Namespace, see the Prometheus Operator documentation.

  2. Verify that the Prometheus Operator installation succeeded using kubectl:

    kubectl get deploy

    NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-operator   1/1     1            1           3m21s

With Prometheus Operator installed, you can move on to configuring RBAC permissions for the Prometheus server.

Configure Prometheus RBAC Permissions

Before rolling out Prometheus, you’ll configure its RBAC privileges using a ClusterRole, and bind this ClusterRole to a ServiceAccount using a ClusterRoleBinding object.

Prometheus needs Kubernetes API access to discover targets and pull ConfigMaps. To learn more about permissions granted in this section, see RBAC from the Prometheus Operator docs.

  1. Create a directory in which you’ll store any Kubernetes manifests used for this guide, and cd into it:

    mkdir operator_k8s
cd operator_k8s

  2. Create a manifest file called prom_rbac.yaml using an editor and paste in the following Kubernetes manifest:

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default

    This creates a ServiceAccount called prometheus and binds it to the prometheus ClusterRole. The manifest grants the ClusterRole get, list, and watch Kubernetes API privileges.

  3. When you’re done editing the manifest, save and close it.

  4. Create the objects using kubectl:

    kubectl apply -f

    serviceaccount/prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created

Now that Prometheus has Kubernetes API access, you can deploy it into the cluster.

Deploy Prometheus

In this step, you’ll launch a 2-replica HA Prometheus deployment into your Kubernetes cluster using a Prometheus resource defined by Prometheus Operator.

This Prometheus resource encodes domain-specific Prometheus configuration into a set of readily configurable YAML fields. Instead of having to manage Prometheus configuration files and learn Prometheus configuration syntax, you can toggle many important configuration parameters by modifying Prometheus object variables in a Kubernetes manifest.

  1. Create a file called prometheus.yaml and paste in the following manifest:

    apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  image: quay.io/prometheus/prometheus:v2.22.1
  nodeSelector:
    kubernetes.io/os: linux
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus
  version: v2.22.1
  serviceMonitorSelector: {}

    Notice that kind is set to Prometheus and not Deployment or Pod. The Prometheus resource abstracts away the underlying controllers and ConfigMaps into streamlined objects used to manipulate Prometheus infrastructure components.

    The manifest configures the following:

    • Names the Prometheus resource prometheus.
    • Gives it an app: prometheus label.
    • Sets the container image used to run Prometheus.
    • Restricts its deployment to Linux nodes.
    • Ensures each replica has 400Mi of memory available to it.
    • Configures its Security Context. To learn more, see Configure a Security Context for a Pod or Container.
    • Sets the ServiceAccount it will use.
    • Sets the Prometheus version.
    • Instructs Prometheus to automatically pick up all configured ServiceMonitor resources using {}. In this guide, you’ll create a ServiceMonitor to get Prometheus to scrape its own metrics.

  2. When you’re done, save and close the file.

  3. Deploy the manifest into your cluster using kubectl apply -f:

    kubectl apply -f

    prometheus.monitoring.coreos.com/prometheus created

    Note: You’ve deployed Prometheus into the default Namespace. To deploy Prometheus in another Namespace, use the -n namespace_name flag with kubectl or set the namespace field for the resource in a Kubernetes manifest file.

  4. Verify the deployment using kubectl get:

    kubectl get prometheus

    NAME         VERSION   REPLICAS   AGE
prometheus   v2.22.1   2          32s

  5. Check the underlying Pods using get pod:

    kubectl get pod

    NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-79cd654746-mdfp6   1/1     Running   0          33m
prometheus-prometheus-0                2/2     Running   1          57s
prometheus-prometheus-1                2/2     Running   1          57s

Now that Prometheus is up and running in our cluster, you can expose it using a service.

Create a Prometheus service

To create the Prometheus service:

  1. Open a manifest file called prom_svc.yaml and paste in the following definitions:

    apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  ports:
  - name: web
    port: 9090
    targetPort: web
  selector:
    app.kubernetes.io/name: prometheus
  sessionAffinity: ClientIP

    This manifest configures the following:

    • Sets the service name to prometheus.
    • Creates an app: prometheus Service label.
    • Exposes port 9090 on a cluster-wide stable IP address and forwards it to the two Prometheus Pods at their default web port (9090).
    • Selects the Prometheus Pods as targets using the app: prometheus label.
    • Uses sessionAffinity: ClientIP to ensure that connections from a particular client get forwarded to the same Pod.

  2. When you’re done, save and close the file.

  3. Create the service using kubectl apply -f:

    kubectl apply -f prom_svc.yaml

    service/prometheus created

  4. Check your work using kubectl get:

    kubectl get service

    NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes            ClusterIP   10.245.0.1       <none>        443/TCP    27h
prometheus            ClusterIP   10.245.106.105   <none>        9090/TCP   26h
prometheus-operated   ClusterIP   None             <none>        9090/TCP   8m52s
prometheus-operator   ClusterIP   None             <none>        8080/TCP   41m

    The prometheus service is up and running.

  5. Access the Prometheus server by forwarding a local port to the Prometheus service running inside of the Kubernetes cluster:

    kubectl port-forward svc/prometheus 9090

    Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

  6. Navigate to http://localhost:9090 to access the Prometheus interface:

    Prometheus UI

  7. Click Status then Targets to see any configured scrape targets.

    This should be empty, as you haven’t yet configured anything to scrape. You’ll do this in the next step.

Create a Prometheus ServiceMonitor

In this step, you’ll create a ServiceMonitor so that the Prometheus server scrapes its own metrics endpoint.

A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource.

Instead of having to modify a Prometheus configuration file, you update a ConfigMap object, and roll out the new configuration, so that Prometheus Operator automatically hooks in new ServiceMonitors to your running Prometheus deployment.

To create the ServiceMonitor:

  1. Create a file called prometheus_servicemonitor.yaml and paste in the following ServiceMonitor resource definition:

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-self
  labels:
    app: prometheus
spec:
  endpoints:
  - interval: 30s
    port: web
  selector:
    matchLabels:
      app: prometheus

    The resource definition configures the following:

    • Names the ServiceMonitor prometheus-self.
    • Gives it an app: prometheus label.
    • Sets the scrape interval to 30s, and scrapes the web port (defined in the prometheus Service).
    • Selects the prometheus Service to scrape using the matchLabels selector with app: prometheus.

  2. When you’re done, save and close the file.

  3. Deploy the using kubectl apply -f:

    kubectl apply -f prometheus_servicemonitor.yaml

    servicemonitor.monitoring.coreos.com/prometheus-self created

    Prometheus Operator should update your Prometheus configuration immediately. In some cases, you may need to wait a minute or so for changes to propagate.

Verify your updates

Verify your changes by forwarding a port to your Prometheus server and checking its configuration:

  1. Use port-forward by running this command:

    kubectl port-forward svc/prometheus 9090

  2. Navigate to Status and then Targets in the Prometheus interface:

    Prometheus Targets

    You should see the two Prometheus replicas as scrape targets.

  3. Navigate to Graph to test metrics collection:

    Prometheus Graph UI

  4. In the Expression box, type prometheus_http_requests_total and hit ENTER.

    You should see a list of scraped metrics and their values. These are HTTP request counts for various Prometheus server endpoints.

    If you’re having trouble configuring a ServiceMonitor, see Troubleshooting ServiceMonitor Changes.

At this point, you’ve configured Prometheus to scrape itself and store metrics locally. In the following steps, you’ll configure Prometheus to ship these metrics to Grafana Cloud.

Create a Kubernetes Secret to store Grafana Cloud credentials

Before configuring Prometheus’s remote_write feature to ship metrics to Grafana Cloud, you’ll need to create a Kubernetes Secret to store your Grafana Cloud Metrics username and password.

To create the Secret:

  1. Find your username by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.

    Your password corresponds to the API key that you generated in the prerequisites section. You can also generate one by clicking Generate now in this panel. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key

    Once you’ve noted your Cloud Prometheus username and password, you’ll be able to create the Kubernetes Secret using a manifest file or directly using kubectl. In this guide, you’ll create it directly using kubectl. To learn more about Kubernetes Secrets, refer to Secrets from the Kubernetes docs.

  2. Run the following command to create a Secret called kubepromsecret:

    kubectl create secret generic kubepromsecret \
  --from-literal=username=<your_grafana_cloud_prometheus_username>\
  --from-literal=password='<your_grafana_cloud_API_key>'

    Note: If you deployed your monitoring stack in a namespace other than default, append the -n flag with the appropriate namespace to the above command.

    To learn more about this command, see Managing Secrets using kubectl from the official Kubernetes documentation.

Now that you’ve created a Secret to store your Grafana Cloud credentials, you can move on to modifying your Prometheus configuration.

Configure Prometheus remote_write and metrics deduplication

In this step, you’ll configure remote_write to ship cluster metrics to Grafana Cloud and to deduplicate metrics.

Prometheus’s remote_write feature allows you to ship metrics to remote endpoints for long-term storage and aggregation. Grafana Cloud’s deduplication feature allows you to deduplicate metrics sent from high-availability Prometheus pairs, reducing your active series usage.

You’ll enable remote_write by modifying the prometheus resource created earlier.

To configure Prometheus remote_write and metrics deduplication:

  1. Open the prometheus.yaml manifest using your editor:

    apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  image: quay.io/prometheus/prometheus:v2.22.1
  nodeSelector:
    kubernetes.io/os: linux
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus
  version: v2.22.1
  serviceMonitorSelector: {}

  2. Add the following section to the bottom of the resource definition:

    . . .
  remoteWrite:
  - url: "<Your Metrics instance remote_write endpoint>"
    basicAuth:
      username:
        name: kubepromsecret
        key: username
      password:
        name: kubepromsecret
        key: password
  replicaExternalLabelName: "__replica__"
  externalLabels:
    cluster: "<choose_a_prom_cluster_name>"

    The resource definition configures the following:

    • Sets the remote_write URL corresponding to Grafana Cloud’s Prometheus metrics endpoint. You can find the /api/prom/push URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal.
    • Configures a basicAuth username and password referencing the Secret created in the earlier step named kubepromsecret.
    • Selects the username and password keys of this Secret.
    • Configures Grafana Cloud metrics deduplication using the replicaExternalLabelName and externalLabels fields.
    • Sets cluster to a value that identifies your Prometheus HA cluster. To learn more, see Deduplicating metrics data sent from high-availability Prometheus pairs.

  3. Save and close the file when you’re done editing.

  4. Roll out the changes using kubectl apply -f:

    kubectl apply -f prometheus.yaml

    prometheus.monitoring.coreos.com/prometheus configured

    At this point, you’ve successfully configured your Prometheus instances to remote_write scraped metrics to Grafana Cloud.

  5. Verify that your changes have propagated to your running Prometheus instances using port-forward:

    kubectl port-forward svc/prometheus 9090

  6. Navigate to http://localhost:9090 in your browser, and then Status and Configuration. Verify that the remote_write and external_labels blocks you appended earlier have propagated to your running Prometheus instances.

    It may take a minute or two for Prometheus Operator to pick up the new configuration.

In the final step, you’ll query your cluster metrics in Grafana Cloud.

Access your Prometheus metrics in Grafana Cloud

Now that your Prometheus instances ship their scraped metrics to Grafana Cloud, you can query and visualize these metrics from the Grafana Cloud platform.

  1. From the Cloud Portal, click Log In next to the Grafana card to log in to Grafana.

  2. Click Explore in the left-side menu.

  3. In the PromQL query box, enter the same metric you tested earlier, prometheus_http_requests_total, and hit SHIFT + ENTER.

    You should see a graph of time series data corresponding to different labels of the prometheus_http_requests_total metric. Grafana queries this data from the Grafana Cloud Metrics data store and not your local cluster.

From here you can create dashboards and panels to visualize and alert on this data. To learn more, see Dashboard overview and Panel overview from the Grafana documentation.

Summary

In this guide, you installed Prometheus Operator into a Kubernetes cluster, deployed a 2-replica HA Prometheus setup into the cluster, and configured it to scrape its own /metrics endpoint. You then configured Prometheus to ship this data to Grafana Cloud and deduplicate the two sets of series to reduce metrics usage.

From here you can add additional endpoints to scrape, like the Kubernetes API, or kubelet metrics from the Kubernetes nodes. To see a fully configured Prometheus Kubernetes stack in action, see the kube-prometheus project.