Configure Kubernetes Monitoring

Use the Kubernetes Monitoring interface to set up monitoring of your Kubernetes cluster and installing preconfigured dashboards and alerts.

Notes:

• You must have the Admin role to install dashboards and alerts.
• If you don’t want to use the preconfigured Agent manifests, you can manually deploy and scrape kube-state-metrics. However, if you don’t use Grafana Agent, you cannot monitor events.

Deploy Kubernetes Monitoring

1. Navigate to your Grafana Cloud instance.

2. Click the Kubernetes Monitoring icon (ship wheel).

   

3. Click Start sending data.

   

4. Select whether you want to use Grafana Agent to send telemetry data to Grafana cloud or to manually send kube-state-metrics.

   Follow the instructions to bring your Kubernetes data into Grafana Cloud.

   The instructions explain how to:

   • Configure using Grafana Agent or Grafana Agent Operator, or by manually sending kube-state-metrics.

   • Install the preconfigured dashboards and alerts.

   • Optionally, configure to send logs.

5. Explore your Kubernetes infrastructure.

Note: If you’ve installed the dashboards and alerts, but haven’t deployed and configured Agent to scrape metrics and collect logs, the Agent configuration instructions will show you how to deploy the following:

• Grafana Agent single-replica StatefulSet that will collect Prometheus metrics & Kubernetes events from objects in your cluster.
• Kube-state-metrics Helm chart (which deploys a KSM Deployment and Service, along with some other access control objects).
• Grafana Agent DaemonSet that will collect logs from Pods in your cluster.

kube-state-metrics

Note that the following metrics are required to use the Kubernetes Monitoring Cluster navigation feature:

- kube_namespace_status_phase
- container_cpu_usage_seconds_total
- kube_pod_status_phase
- kube_pod_start_time
- kube_pod_container_status_restarts_total
- kube_pod_container_info
- kube_pod_container_status_waiting_reason
- kube_daemonset.\*
- kube_replicaset.\*
- kube_statefulset.\*
- kube_job.\*
- kube_node*
- kube_cluster*
- node_cpu_seconds_total
- node_memory_MemAvailable_bytes
- node_filesystem_size_bytes
- node_namespace_pod_container
- container_memory_working_set_bytes
- job="integrations/kubernetes/eventhandler" (for event logs, comes default with Grafana agent)

NOTE: Logs are not required for Kubernetes Monitoring to work, but they provide additional context in some views of the Cluster Navigation tab. Log entries must be shipped to a Loki data source with cluster, namespace, and pod labels.

Configured scrape targets

If you configure Kubernetes Monitoring using Grafana Agent, by default, Agent scrapes the following targets:

• cAdvisor (one per node): cAdvisor is present on each node in your cluster and emits container resource usage metrics like CPU usage, memory usage, and disk usage.
• kubelet (one per node): kubelet is present on each node and emits metrics specific to the kubelet process like kubelet_running_pods and kubelet_running_container_count.
• kube-state-metrics (one replica, by default): kube-state-metrics runs as a Deployment and Service in your cluster and emits Prometheus metrics that track the state of objects in your cluster, like Pods, Deployments, DaemonSets, and more.

The default ConfigMap configures an allowlist to drop all metrics not referenced in the Kubernetes Monitoring dashboards, alerts, and recording rules. You can optionally modify this allowlist, replace it with a denylist (by using the drop directive), omit it entirely, or move it to the remote_write level so that it applies globally to all configured scrape jobs. To learn more, see Reducing Prometheus metrics usage with relabeling.

Scrape application Pod metrics

Kubenetes Monitoring does not scrape application Prometheus metrics by default, but you can configure Grafana Agent to also scrape application Prometheus metrics, like those available at the standard /metrics endpoint on pods.

To add a scrape job targeting all /metrics endpoints on your cluster pods, do the following:

1. Add the following to the bottom of your Agent scrape config:

   . . .
   - job_name: "kubernetes-pods"
       kubernetes_sd_configs:
         - role: pod
       relabel_configs:
         # Example relabel to scrape only Pods that have
         # "example.io/should_be_scraped = true" annotation.
         #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
         #    action: keep
         #    regex: true
         #
         # Example relabel to customize metric path based on Pod
         # "example.io/metric_path = <metric path>" annotation.
         #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
         #    action: replace
         #    target_label: __metrics_path__
         #    regex: (.+)
         #
         # Example relabel to scrape only single, desired port for the Pod
         # based on Pod "example.io/scrape_port = <port>" annotation.
         #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
         #    action: replace
         #    regex: ([^:]+)(?::\d+)?;(\d+)
         #    replacement: $1:$2
         #    target_label: __address__
         # Expose Pod labels as metric labels
         - action: labelmap
           regex: __meta_kubernetes_pod_label_(.+)
         # Expose Pod namespace as metric namespace label
         - source_labels: [__meta_kubernetes_namespace]
           action: replace
           target_label: namespace
         # Expose Pod name as metric name label
         - source_labels: [__meta_kubernetes_pod_name]
           action: replace
           target_label: pod
   

   This config adds every defined Pod container port to Agent’s scrape targets, discovered using Agent’s Kubernetes service discovery mechanism. You can optionally uncomment the relevant sections to customize the metrics path (the default is /metrics), specify a sample port, or use Pod annotations to declaratively specify which targets Agent should scrape in your Pod manifests. To learn more, see the examples in the official Prometheus project repo.

   To learn more about configuring Agent, see Configure Grafana Agent from the Agent docs. To learn more about available kubernetes_sd_configs labels and roles (we used the pod role here), see kubernetes_sd_config from the Prometheus docs.

2. Deploy the updated config into your cluster using kubectl apply -f:

   kubectl apply -f <YOUR-CONFIGMAP>.yaml
   

3. Restart the agent to pick up the config changes.

   kubectl rollout restart deployment/grafana-agent
   

For a complete example, see Monitor an app on Kubernetes using Grafana Agent.

Install manually

You can manually connect your Kubernetes data to Grafana Cloud rather than using the manifests provided in the Kubernetes Monitoring configuration using one of the following methods:

• Directly deploy Grafana Agent for metrics, logs, and traces.

• Use Kubernetes Agent Operator (beta) to deploy and configure Grafana Agent using a Prometheus-style operator and Kubernetes custom resources.

• Use remote_write to ship Prometheus metrics if you have an existing Prometheus deployment, and want to ship your Kubernetes metrics to Grafana Kubernetes Monitoring.

Reinstall or upgrade Kubernetes Monitoring

Grafana Agent, dashboards, alerting rules, recording rules, kube-state-metrics, and Kubernetes manifests are updated regularly. You must update these components manually to take advantage of any updates. See how to update Kubernetes Monitoring components.