Install Prometheus Operator with Grafana Cloud for Kubernetes
In this guide, you’ll learn how to install Prometheus Operator in a Kubernetes (K8s) cluster, configure it to scrape an endpoint, and ship scraped metrics to Grafana Cloud.
Prometheus Operator implements the Kubernetes Operator pattern for managing a Prometheus-based Kubernetes monitoring stack. A Kubernetes Operator consists of Kubernetes custom resources and Kubernetes controller code. Together, these abstract away the management and implementation details of running a given service on Kubernetes. To learn more about Kubernetes Operators, see the Operator pattern from the Kubernetes docs.
The Prometheus Operator installs a set of Kubernetes custom resources that simplify Prometheus deployment and configuration. For example, using the ServiceMonitor
custom resource, you can configure how to monitor Kubernetes services in K8s YAML manifests instead of Prometheus configuration code. The Operator controller then communicates with the K8s API server to add service /metrics
endpoints and automatically generate the required Prometheus scrape configurations for the configured services. To learn more about Prometheus Operator, see the Prometheus Operator GitHub repository.
In this guide, you’ll begin by installing Prometheus Operator into the Kubernetes cluster. Next, you’ll launch a 2-replica high-availability (HA) Prometheus deployment into the cluster using the Operator, and then expose the Prometheus server as a service. Finally, you’ll create a ServiceMonitor to instruct Prometheus to scrape itself, and then ship those scraped metrics to Grafana Cloud.
Before you begin
Before you begin, you should have the following items available to you:
- A Kubernetes
>=1.16.0
cluster with role-based access control (RBAC) enabled. - A Grafana Cloud account.
- A Grafana Cloud API key with the
MetricsPublisher
role. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key. - The
kubectl
command-line tool installed on your local machine and configured to connect to your cluster. See how to installkubectl
.
Install Prometheus Operator
You’ll start by installing Prometheus Operator into the Kubernetes cluster. You’ll install all of Prometheus Operator’s Kubernetes custom resource definitions (CRDs) that define the Prometheus, Alertmanager, and ServiceMonitor abstractions used to configure the monitoring stack. You’ll also deploy a Prometheus Operator controller into the cluster.
Note: Following this guide, you’ll install everything into the default
Namespace. To learn how to install Prometheus Operator into another Namespace, see the Prometheus Operator documentation.
Install the Operator using the
bundle.yaml
file in the Prometheus Operator GitHub repository:kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
You should see the following output:
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created clusterrole.rbac.authorization.k8s.io/prometheus-operator created deployment.apps/prometheus-operator created serviceaccount/prometheus-operator created service/prometheus-operator created
bundle.yaml
installs CRDs for Prometheus objects as well as a Prometheus Operator controller and service.Note: Following this guide, you’ll install everything into the
default
Namespace. To learn how to install Prometheus Operator into another Namespace, see the Prometheus Operator documentation.Verify that the Prometheus Operator installation succeeded using
kubectl
:kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 3m21s
With Prometheus Operator installed, you can move on to configuring RBAC permissions for the Prometheus server.
Configure Prometheus RBAC Permissions
Before rolling out Prometheus, you’ll configure its RBAC privileges using a ClusterRole, and bind this ClusterRole to a ServiceAccount using a ClusterRoleBinding object.
Prometheus needs Kubernetes API access to discover targets and pull ConfigMaps. To learn more about permissions granted in this section, see RBAC from the Prometheus Operator docs.
Create a directory in which you’ll store any Kubernetes manifests used for this guide, and
cd
into it:mkdir operator_k8s cd operator_k8s
Create a manifest file called
prom_rbac.yaml
using an editor and paste in the following Kubernetes manifest:apiVersion: v1 kind: ServiceAccount metadata: name: prometheus --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/metrics - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: [""] resources: - configmaps verbs: ["get"] - apiGroups: - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: default
This creates a ServiceAccount called
prometheus
and binds it to theprometheus
ClusterRole. The manifest grants the ClusterRoleget
,list
, andwatch
Kubernetes API privileges.When you’re done editing the manifest, save and close it.
Create the objects using
kubectl
:kubectl apply -f
serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created
Now that Prometheus has Kubernetes API access, you can deploy it into the cluster.
Deploy Prometheus
In this step, you’ll launch a 2-replica HA Prometheus deployment into your Kubernetes cluster using a Prometheus
resource defined by Prometheus Operator.
This Prometheus
resource encodes domain-specific Prometheus configuration into a set of readily configurable YAML fields. Instead of having to manage Prometheus configuration files and learn Prometheus configuration syntax, you can toggle many important configuration parameters by modifying Prometheus
object variables in a Kubernetes manifest.
Create a file called
prometheus.yaml
and paste in the following manifest:apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus labels: app: prometheus spec: image: quay.io/prometheus/prometheus:v2.22.1 nodeSelector: kubernetes.io/os: linux replicas: 2 resources: requests: memory: 400Mi securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus version: v2.22.1 serviceMonitorSelector: {}
Notice that
kind
is set toPrometheus
and notDeployment
orPod
. ThePrometheus
resource abstracts away the underlying controllers and ConfigMaps into streamlined objects used to manipulate Prometheus infrastructure components.The manifest configures the following:
- Names the
Prometheus
resourceprometheus
. - Gives it an
app: prometheus
label. - Sets the container image used to run Prometheus.
- Restricts its deployment to Linux nodes.
- Ensures each replica has
400Mi
of memory available to it. - Configures its Security Context. To learn more, see Configure a Security Context for a Pod or Container.
- Sets the ServiceAccount it will use.
- Sets the Prometheus version.
- Instructs Prometheus to automatically pick up all configured
ServiceMonitor
resources using{}
. In this guide, you’ll create aServiceMonitor
to get Prometheus to scrape its own metrics.
- Names the
When you’re done, save and close the file.
Deploy the manifest into your cluster using
kubectl apply -f
:kubectl apply -f
prometheus.monitoring.coreos.com/prometheus created
Note: You’ve deployed Prometheus into the
default
Namespace. To deploy Prometheus in another Namespace, use the-n namespace_name
flag withkubectl
or set thenamespace
field for the resource in a Kubernetes manifest file.Verify the deployment using
kubectl get
:kubectl get prometheus
NAME VERSION REPLICAS AGE prometheus v2.22.1 2 32s
Check the underlying Pods using
get pod
:kubectl get pod
NAME READY STATUS RESTARTS AGE prometheus-operator-79cd654746-mdfp6 1/1 Running 0 33m prometheus-prometheus-0 2/2 Running 1 57s prometheus-prometheus-1 2/2 Running 1 57s
Now that Prometheus is up and running in our cluster, you can expose it using a service.
Create a Prometheus service
To create the Prometheus service:
Open a manifest file called
prom_svc.yaml
and paste in the following definitions:apiVersion: v1 kind: Service metadata: name: prometheus labels: app: prometheus spec: ports: - name: web port: 9090 targetPort: web selector: app.kubernetes.io/name: prometheus sessionAffinity: ClientIP
This manifest configures the following:
- Sets the service name to
prometheus
. - Creates an
app: prometheus
Service
label. - Exposes port
9090
on a cluster-wide stable IP address and forwards it to the two Prometheus Pods at their defaultweb
port (9090
). - Selects the Prometheus Pods as targets using the
app: prometheus
label. - Uses
sessionAffinity: ClientIP
to ensure that connections from a particular client get forwarded to the same Pod.
- Sets the service name to
When you’re done, save and close the file.
Create the service using
kubectl apply -f
:kubectl apply -f prom_svc.yaml
service/prometheus created
Check your work using
kubectl get
:kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 27h prometheus ClusterIP 10.245.106.105 <none> 9090/TCP 26h prometheus-operated ClusterIP None <none> 9090/TCP 8m52s prometheus-operator ClusterIP None <none> 8080/TCP 41m
The
prometheus
service is up and running.Access the Prometheus server by forwarding a local port to the Prometheus service running inside of the Kubernetes cluster:
kubectl port-forward svc/prometheus 9090
Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090
Navigate to
http://localhost:9090
to access the Prometheus interface:Click Status then Targets to see any configured scrape targets.
This should be empty, as you haven’t yet configured anything to scrape. You’ll do this in the next step.
Create a Prometheus ServiceMonitor
In this step, you’ll create a ServiceMonitor so that the Prometheus server scrapes its own metrics endpoint.
A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource.
Instead of having to modify a Prometheus configuration file, you update a ConfigMap object, and roll out the new configuration, so that Prometheus Operator automatically hooks in new ServiceMonitors to your running Prometheus deployment.
To create the ServiceMonitor:
Create a file called
prometheus_servicemonitor.yaml
and paste in the following ServiceMonitor resource definition:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: prometheus-self labels: app: prometheus spec: endpoints: - interval: 30s port: web selector: matchLabels: app: prometheus
The resource definition configures the following:
- Names the ServiceMonitor
prometheus-self
. - Gives it an
app: prometheus
label. - Sets the scrape interval to
30s
, and scrapes theweb
port (defined in theprometheus
Service
). - Selects the
prometheus
Service
to scrape using thematchLabels
selector withapp: prometheus
.
- Names the ServiceMonitor
When you’re done, save and close the file.
Deploy the using
kubectl apply -f
:kubectl apply -f prometheus_servicemonitor.yaml
servicemonitor.monitoring.coreos.com/prometheus-self created
Prometheus Operator should update your Prometheus configuration immediately. In some cases, you may need to wait a minute or so for changes to propagate.
Verify your updates
Verify your changes by forwarding a port to your Prometheus server and checking its configuration:
Use
port-forward
by running this command:kubectl port-forward svc/prometheus 9090
Navigate to Status and then Targets in the Prometheus interface:
You should see the two Prometheus replicas as scrape targets.
Navigate to Graph to test metrics collection:
In the Expression box, type
prometheus_http_requests_total
and hitENTER
.You should see a list of scraped metrics and their values. These are HTTP request counts for various Prometheus server endpoints.
If you’re having trouble configuring a ServiceMonitor, see Troubleshooting ServiceMonitor Changes.
At this point, you’ve configured Prometheus to scrape itself and store metrics locally. In the following steps, you’ll configure Prometheus to ship these metrics to Grafana Cloud.
Create a Kubernetes Secret to store Grafana Cloud credentials
Before configuring Prometheus’s remote_write
feature to ship metrics to Grafana Cloud, you’ll need to create a Kubernetes Secret to store your Grafana Cloud Metrics username and password.
To create the Secret:
Find your username by navigating to your stack in the Cloud Portal and clicking Details next to the Prometheus panel.
Your password corresponds to the API key that you generated in the prerequisites section. You can also generate one by clicking Generate now in this panel. To learn how to create a Grafana Cloud API key, see Create a Grafana Cloud API key
Once you’ve noted your Cloud Prometheus username and password, you’ll be able to create the Kubernetes Secret using a manifest file or directly using
kubectl
. In this guide, you’ll create it directly usingkubectl
. To learn more about Kubernetes Secrets, refer to Secrets from the Kubernetes docs.Run the following command to create a Secret called
kubepromsecret
:kubectl create secret generic kubepromsecret \ --from-literal=username=<your_grafana_cloud_prometheus_username>\ --from-literal=password='<your_grafana_cloud_API_key>'
Note: If you deployed your monitoring stack in a namespace other than
default
, append the-n
flag with the appropriate namespace to the above command.To learn more about this command, see Managing Secrets using kubectl from the official Kubernetes documentation.
Now that you’ve created a Secret to store your Grafana Cloud credentials, you can move on to modifying your Prometheus configuration.
Configure Prometheus remote_write and metrics deduplication
In this step, you’ll configure remote_write
to ship cluster metrics to Grafana Cloud and to deduplicate metrics.
Prometheus’s remote_write
feature allows you to ship metrics to remote endpoints for long-term storage and aggregation. Grafana Cloud’s deduplication feature allows you to deduplicate metrics sent from high-availability Prometheus pairs, reducing your active series usage.
You’ll enable remote_write
by modifying the prometheus
resource created earlier.
To configure Prometheus remote_write and metrics deduplication:
Open the
prometheus.yaml
manifest using your editor:apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus labels: app: prometheus spec: image: quay.io/prometheus/prometheus:v2.22.1 nodeSelector: kubernetes.io/os: linux replicas: 2 resources: requests: memory: 400Mi securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: prometheus version: v2.22.1 serviceMonitorSelector: {}
Add the following section to the bottom of the resource definition:
. . . remoteWrite: - url: "<Your Metrics instance remote_write endpoint>" basicAuth: username: name: kubepromsecret key: username password: name: kubepromsecret key: password replicaExternalLabelName: "__replica__" externalLabels: cluster: "<choose_a_prom_cluster_name>"
The resource definition configures the following:
- Sets the
remote_write
URL corresponding to Grafana Cloud’s Prometheus metrics endpoint. You can find the/api/prom/push
URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal. - Configures a
basicAuth
username and password referencing the Secret created in the earlier step namedkubepromsecret
. - Selects the
username
andpassword
keys of this Secret. - Configures Grafana Cloud metrics deduplication using the
replicaExternalLabelName
andexternalLabels
fields. - Sets
cluster
to a value that identifies your Prometheus HA cluster. To learn more, see Deduplicating metrics data sent from high-availability Prometheus pairs.
- Sets the
Save and close the file when you’re done editing.
Roll out the changes using
kubectl apply -f
:kubectl apply -f prometheus.yaml
prometheus.monitoring.coreos.com/prometheus configured
At this point, you’ve successfully configured your Prometheus instances to
remote_write
scraped metrics to Grafana Cloud.Verify that your changes have propagated to your running Prometheus instances using
port-forward
:kubectl port-forward svc/prometheus 9090
Navigate to
http://localhost:9090
in your browser, and then Status and Configuration. Verify that theremote_write
andexternal_labels
blocks you appended earlier have propagated to your running Prometheus instances.It may take a minute or two for Prometheus Operator to pick up the new configuration.
In the final step, you’ll query your cluster metrics in Grafana Cloud.
Access your Prometheus metrics in Grafana Cloud
Now that your Prometheus instances ship their scraped metrics to Grafana Cloud, you can query and visualize these metrics from the Grafana Cloud platform.
From the Cloud Portal, click Log In next to the Grafana card to log in to Grafana.
Click Explore in the left-side menu.
In the PromQL query box, enter the same metric you tested earlier,
prometheus_http_requests_total
, and hitSHIFT + ENTER
.You should see a graph of time series data corresponding to different labels of the
prometheus_http_requests_total
metric. Grafana queries this data from the Grafana Cloud Metrics data store and not your local cluster.
From here you can create dashboards and panels to visualize and alert on this data. To learn more, see Dashboard overview and Panel overview from the Grafana documentation.
Summary
In this guide, you installed Prometheus Operator into a Kubernetes cluster, deployed a 2-replica HA Prometheus setup into the cluster, and configured it to scrape its own /metrics
endpoint. You then configured Prometheus to ship this data to Grafana Cloud and deduplicate the two sets of series to reduce metrics usage.
From here you can add additional endpoints to scrape, like the Kubernetes API, or kubelet
metrics from the Kubernetes nodes. To see a fully configured Prometheus Kubernetes stack in action, see the kube-prometheus project.