Configure Kubernetes Monitoring using Grafana Agent

This guide shows how to configure Kubernetes Monitoring using Grafana Agent to scrape telemetry data.

When you configure Kubernetes Monitoring using Grafana Agent, Agent scrapes the following targets by default:

cAdvisor (one per node): cAdvisor is present on each node in your cluster and emits container resource usage metrics like CPU usage, memory usage, and disk usage.
kubelet (one per node): kubelet is present on each node and emits metrics specific to the kubelet process like kubelet_running_pods and kubelet_running_container_count.
kube-state-metrics (one replica, by default): kube-state-metrics runs as a Deployment and Service in your cluster and emits Prometheus metrics that track the state of objects in your cluster, like Pods, Deployments, and DaemonSets.

The default ConfigMap configures an allowlist to drop all metrics not referenced in the Kubernetes Monitoring dashboards, alerts, and recording rules. You can optionally modify this allowlist, replace it with a denylist (by using the drop directive), omit it entirely, or move it to the remote_write level so that it applies globally to all configured scrape jobs. To learn more, see Reducing Prometheus metrics usage with relabeling. See also Reducing Kubernetes metrics usage for other methods of controlling usage.

Before you start

To deploy Kubernetes Monitoring, you need:

A Kubernetes cluster, environment, or fleet you want to monitor
The kubectl, curl, and envsubst command-line tools

Deploy Kubernetes Monitoring using Grafana Agent

This section describes how to deploy Kubernetes Monitoring by installing the following:

Grafana Agent single-replica StatefulSet, which collects Prometheus metrics and Kubernetes events from objects in your cluster.
kube-state-metrics Helm chart, which deploys a kube-state-metrics (KSM) Deployment and Service, along with some other access control objects.
Grafana Agent DaemonSet, which collects logs from Pods in your cluster.

Important: You should have only one job scraping kube-state-metrics. If you have multiple scrape jobs running at the same time, you might see an error similar to the following when you try to view objects in Cluster navigation: execution: found duplicate series for the match group...

To deploy Kubernetes Monitoring using Grafana Agent:

Navigate to your Grafana Cloud instance, click Observability in the left-side menu and select Kubernetes.
Click Start sending data.
Kubernetes Monitoring displays the Configuration tab.
Select a Prometheus data source and, optionally, a Loki data source if you want to send log data.

Install preconfigured dashboards and alerts

Note: You must have the Admin role to install dashboards and alerts.

To install the preconfigured dashboards and alerts:

Click Install dashboards and rules.
Kubernetes Monitoring installs the preconfigured dashboards and rules. After you set up Kubernetes Monitoring, you can select the Dashboards tab to view your telemetry data.

Configure Grafana Agent for metrics

To configure Grafana Agent for Kubernetes Monitoring:

Under Configuration Instructions, click Agent.
Click Agent configuration instructions.
Under Prerequisites (step 1), check that you have met the prerequisites and follow the instructions to complete them, if necessary.
Under Metrics & Events (step 2), in the Create Agent ConfigMap section, replace default in the Namespace field with the Namespace that you want to monitor in your cluster.
Under the ConfigMap box, click Copy to clipboard and run it in your terminal.
The Grafana Agent ConfigMap is generated for your Namespace:
```
configmap/grafana-agent created
```
The ConfigMap configures Grafana Agent to scrape cadvisor, kubelet, and kube-state-metrics, and to collect events to be shipped to Grafana Loki.
In the Create Agent StatefulSet section, click Copy to clipboard beneath the manifest box and run it in your terminal.
The manifest uses the same Namespace you set in the ConfigMap, installs a Grafana Agent StatefulSet into your cluster, and configures role-based access control (RBAC) permissions:
```
clusterrole.rbac.authorization.k8s.io/grafana-agent configured
clusterrolebinding.rbac.authorization.k8s.io/grafana-agent created
service/grafana-agent created
statefulsets.apps/grafana-agent created
```
In the Deploy kube-state-metrics section, click Copy to clipboard beneath the box containing the helm repo add command and run it in your terminal.
kube-state-metrics is a service that listens to the Kubernetes API server and generates metrics on the state of the objects. The metrics are exported on the HTTP endpoint /metrics on the listening port.
To verify the step, enter kubectl get pods -w in your terminal. You should see a pod with a name that starts with `ksm-kube-state-metrics-"
If you want to install kube-state-metrics without using Helm, you can install manually by using the example manifests in the kube-state-metrics repo. See the list of kube-state-metrics that are scraped by default.
To deploy Node Exporter (required for resource efficiency monitoring), under Deploy node_exporter, run the commands from the clipboard in your terminal.
These commands install Node Exporter as a DaemonSet in your cluster.

Configure Grafana Agent for logs

To configure logging:

Under Logs: Deploy Agent ConfigMap & DaemonSet, click Copy to clipboard and run it in your terminal.
The ConfigMap is created:
```
configmap/grafana-agent-logs created
```
The grafana-agent-logs ConfigMap configures the Grafana Agent DaemonSet to tail container logs and ship them to Grafana Cloud.
If your Kubernetes cluster doesn’t use Docker, copy the script to an editor and replace docker: {}' with 'cri: {}. If you want to have your log lines annotated with colors by log level in the explore panel please modify the config map as follows:
Append - regex: expression: "(?P<level>[a-zA-Z]+):(.*)"
- labels:
level:
After docker: {}' or 'cri: {} (the regex and docker must have the same indention)
To install the Grafana Agent Daemonset, under the logs manifest box (starts with MANIFEST_URL), click Copy to clipboard and run it in your terminal.
This manifest installs an Agent DaemonSet into your cluster and configures RBAC permissions.
To verify the steps, enter kubectl get pods -w in your terminal. You should see a pod with a name that starts with `grafana-agent-logs-"

Complete the configuration

To finish up:

Click Test and finish configuration.
Grafana Agent begins collecting your Kubernetes telemetry data.
Explore your deployment:
- In the Cluster navigation tab, click your Namespace to view the grafana-agent StatefulSet, the grafana-agent-logs DaemonSet, and the ksm-kube-state-metrics deployment.
- Click the kube-system Namespace to see Kubernetes-specific resources.
- In the Nodes tab, click into the Nodes of your cluster to view their condition, utilization, and pod density.

Next steps

Explore your Kubernetes infrastructure to view the monitoring data.

If you find that you need to update your configuration, see Update Grafana Kubernetes components.

To monitor an app in your Kubernetes cluster, see the Monitor an app with Kubernetes Monitoring tutorial. You will learn how to deploy an instrumented three-tier (data layer, app logic layer, load-balancing layer) web application into a Kubernetes cluster, and leverage Kubernetes Monitoring to monitor the application.