Configure Kubernetes Monitoring
Use the Kubernetes Monitoring interface to set up monitoring of your Kubernetes cluster and installing preconfigured dashboards and alerts.
- You must have the Admin role to install dashboards and alerts.
- If you don’t want to use the preconfigured Agent manifests, you can manually deploy and scrape kube-state-metrics. However, if you don’t use Grafana Agent, you cannot monitor events.
Deploy Kubernetes Monitoring
Navigate to your Grafana Cloud instance.
Click the Kubernetes Monitoring icon (ship wheel).
Click Start sending data.
Select whether you want to use Grafana Agent to send telemetry data to Grafana cloud or to manually send kube-state-metrics.
Follow the instructions to bring your Kubernetes data into Grafana Cloud.
The instructions explain how to:
Configure using Grafana Agent or Grafana Agent Operator, or by manually sending kube-state-metrics.
Install the preconfigured dashboards and alerts.
Optionally, configure to send logs.
Note: If you’ve installed the dashboards and alerts, but haven’t deployed and configured Agent to scrape metrics and collect logs, the Agent configuration instructions will show you how to deploy the following:
- Grafana Agent single-replica StatefulSet that will collect Prometheus metrics & Kubernetes events from objects in your cluster.
- Kube-state-metrics Helm chart (which deploys a KSM Deployment and Service, along with some other access control objects).
- Grafana Agent DaemonSet that will collect logs from Pods in your cluster.
Note that the following metrics are required to use the Kubernetes Monitoring Cluster navigation feature:
- kube_namespace_status_phase - container_cpu_usage_seconds_total - kube_pod_status_phase - kube_pod_start_time - kube_pod_container_status_restarts_total - kube_pod_container_info - kube_pod_container_status_waiting_reason - kube_daemonset.\* - kube_replicaset.\* - kube_statefulset.\* - kube_job.\* - kube_node* - kube_cluster* - node_cpu_seconds_total - node_memory_MemAvailable_bytes - node_filesystem_size_bytes - node_namespace_pod_container - container_memory_working_set_bytes - job="integrations/kubernetes/eventhandler" (for event logs, comes default with Grafana agent)
NOTE: Logs are not required for Kubernetes Monitoring to work, but they provide additional context in some views of the Cluster Navigation tab. Log entries must be shipped to a Loki data source with
Configured scrape targets
If you configure Kubernetes Monitoring using Grafana Agent, by default, Agent scrapes the following targets:
- cAdvisor (one per node): cAdvisor is present on each node in your cluster and emits container resource usage metrics like CPU usage, memory usage, and disk usage.
- kubelet (one per node): kubelet is present on each node and emits metrics specific to the kubelet process like
- kube-state-metrics (one replica, by default): kube-state-metrics runs as a Deployment and Service in your cluster and emits Prometheus metrics that track the state of objects in your cluster, like Pods, Deployments, DaemonSets, and more.
The default ConfigMap configures an allowlist to drop all metrics not referenced in the Kubernetes Monitoring dashboards, alerts, and recording rules. You can optionally modify this allowlist, replace it with a denylist (by using the
drop directive), omit it entirely, or move it to the
remote_write level so that it applies globally to all configured scrape jobs. To learn more, see Reducing Prometheus metrics usage with relabeling.
Scrape application Pod metrics
Kubenetes Monitoring does not scrape application Prometheus metrics by default, but you can configure Grafana Agent to also scrape application Prometheus metrics, like those available at the standard
/metrics endpoint on pods.
To add a scrape job targeting all
/metrics endpoints on your cluster pods, do the following:
Add the following to the bottom of your Agent scrape config:
. . . - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: # Example relabel to scrape only Pods that have # "example.io/should_be_scraped = true" annotation. # - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped] # action: keep # regex: true # # Example relabel to customize metric path based on Pod # "example.io/metric_path = <metric path>" annotation. # - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path] # action: replace # target_label: __metrics_path__ # regex: (.+) # # Example relabel to scrape only single, desired port for the Pod # based on Pod "example.io/scrape_port = <port>" annotation. # - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port] # action: replace # regex: ([^:]+)(?::\d+)?;(\d+) # replacement: $1:$2 # target_label: __address__ # Expose Pod labels as metric labels - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # Expose Pod namespace as metric namespace label - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace # Expose Pod name as metric name label - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod
This config adds every defined Pod container port to Agent’s scrape targets, discovered using Agent’s Kubernetes service discovery mechanism. You can optionally uncomment the relevant sections to customize the metrics path (the default is
/metrics), specify a sample port, or use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests. To learn more, see the examples in the official Prometheus project repo.
To learn more about configuring Agent, see Configure Grafana Agent from the Agent docs. To learn more about available
kubernetes_sd_configslabels and roles (we used the
podrole here), see kubernetes_sd_config from the Prometheus docs.
Deploy the updated config into your cluster using
kubectl apply -f:
kubectl apply -f <YOUR-CONFIGMAP>.yaml
Restart the agent to pick up the config changes.
kubectl rollout restart deployment/grafana-agent
For a complete example, see Monitor an app on Kubernetes using Grafana Agent.
You can manually connect your Kubernetes data to Grafana Cloud rather than using the manifests provided in the Kubernetes Monitoring configuration using one of the following methods:
Directly deploy Grafana Agent for metrics, logs, and traces.
Use Kubernetes Agent Operator (beta) to deploy and configure Grafana Agent using a Prometheus-style operator and Kubernetes custom resources.
Use remote_write to ship Prometheus metrics if you have an existing Prometheus deployment, and want to ship your Kubernetes metrics to Grafana Kubernetes Monitoring.
Reinstall or upgrade Kubernetes Monitoring
Grafana Agent, dashboards, alerting rules, recording rules, kube-state-metrics, and Kubernetes manifests are updated regularly. You must update these components manually to take advantage of any updates. See how to update Kubernetes Monitoring components.
Related Grafana Cloud resources
How to set up and visualize synthetic monitoring at scale with Grafana Cloud
Learn how to use Kubernetes, Grafana Loki, and Grafana Cloud’s synthetic monitoring feature to set up your infrastructure's checks in this GrafanaCONline session.
Using Grafana Cloud to drive manufacturing plant efficiency
This GrafanaCONline session tells how Grafana helps a 75-year-old manufacturing company with product quality and equipment maintenance.