Documentation Grafana Cloud Observe and act Monitor infrastructure Kubernetes Monitoring Configure Kubernetes Monitoring Configure with Helm chart Configuration steps

Configuration steps for Kubernetes Monitoring with Helm chart

Complete the configuration process on the Cluster configuration tab. The wizard guides you through five steps: Backend and distribution, Access token, Choose setup, Monitoring, and Deployment. Complete each step, then click Next.

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on this Configuration page.

Try it

Before you begin

Make sure you have met the prerequisites required for these configuration steps.

To deploy Kubernetes Monitoring with the Helm chart, you need:

• A Grafana Cloud account
• The Admin role to install alerts
• To have completed the activation of the app
• A Kubernetes Cluster, environment, or fleet you want to monitor
• The kubectl and Helm command-line tools

• To check version compatibility for the features you plan to enable:
  • kube-state-metrics uses client-go to communicate with Clusters. For Kubernetes client-go version compatibility, refer to kube-state-metrics.
  • Cost metrics with OpenCost require Kubernetes 1.8 or later.
  • Visualizations for storage require Helm chart release v1.5.1 or later.

Backend and distribution

Install the required recording and alert rules into your Grafana Cloud instance, and choose the platform for your Cluster.

1. Under Backend Installation, click Install to install the required, preconfigured alert rules and recording rules to your Grafana instance. When they’re installed, the status shows Alert rules and recording rules installed.

2. Under Select your Kubernetes platform, choose your platform:

   • Kubernetes, for EKS on EC2, GKE, and similar distributions
   • Azure AKS
   • GKE Autopilot
   • EKS on Fargate
   • IBM Cloud
   • OpenShift

3. Click Next.

Access token

An access policy token is required for Grafana Alloy, the telemetry collector that the Helm chart deploys on your Cluster, to send metrics and logs to Grafana Cloud. Choose how the Helm chart authenticates to Grafana Cloud. You can create a new access policy token, use an existing token, or point the chart at a stored Kubernetes Secret that already exists in your Cluster. Refer to Grafana Cloud Access Policies for more information.

To create a new token:

1. Click Create a new token.

2. In the Access policy token name box, enter a name. This name helps you view and revoke the token in Grafana.com later.

3. In the Expiration date box, select or enter an expiration in days, or keep No expiry.

   The token uses the predefined scopes metrics:read and set:alloy-data-write. A scope defines which permissions a token has. For more information, refer to Scopes.

4. Click Create token.

To use an existing token:

1. Click Use an existing token.

2. Paste your token into the box.

To use a stored Kubernetes Secret:

1. Click Use a stored Kubernetes Secret.

2. In the Secret name box, enter the name of the Secret that already exists in your Cluster.

3. In the Namespace box, keep the installation namespace or enter the namespace that holds the Secret.

4. Optionally, expand the advanced options to change the Username key and Password key if your Secret doesn’t use the default keys username and password.

   The Secret must hold your Grafana Cloud instance ID as the username and an access policy token as the password, with the metrics:read and set:alloy-data-write scopes. To find your instance ID, refer to Find instance endpoints. The configuration page provides a kubectl command that includes your instance ID, which you can copy to create the Secret. Because the chart reads the token from the Secret at runtime, your token isn’t stored in Grafana and doesn’t appear in the generated Helm command.

5. Click Next.

Choose setup

On the Choose setup step, select a preconfigured Alloy deployment based on the level of observability you need. The screen title reads Choose your Alloy deployment. Each option determines what telemetry you can collect now and what you can enable later without reinstalling. Expand Learn what this deployment includes on any option to see its details. Your choice sets the collector presets in the generated Helm values.

Before you compare the options, it helps to know the terms they use:

• A collector is an instance of Grafana Alloy that runs on every Node (a DaemonSet), gathers telemetry from your Cluster, and sends it to Grafana Cloud. Every option deploys collectors.
• A clustered backend is a second Alloy workload (a Deployment) that runs on your own Nodes and aggregates cluster-wide data. (It’s unrelated to the Backend Installation in the first step, which installs recording and alert rules into your Grafana Cloud instance.)
• The large and xlarge labels are the chart’s size presets. They set the CPU and memory for each instance.
• Autoscaling lets a deployment handle more load without a reinstall. Buffering to persistent storage (a PVC) lets the backend hold data on disk so it survives restarts.

Your selection sets the default monitoring options on the Monitoring step, which you can change there.

Click Next.

Monitoring

On the Monitoring step, enable the remote monitoring features and managed pipelines you want from Grafana Cloud. The screen title reads Remote monitoring and service discovery. The wizard requires the Enable Kubernetes monitoring selection; auto-discovery of services is optional.

1. Select Enable Kubernetes monitoring to instrument your Cluster with the recommended signals for most production Clusters.

   Note

   After you activate Kubernetes monitoring, billing begins based on your host and container hours.

2. Optionally, select Auto-discover services on my cluster to create a managed discovery pipeline in Fleet Management, the Grafana Cloud service for managing Alloy collectors. No application telemetry is collected unless you enable instrumentation. To change whether this pipeline is set up, open the card’s menu, select Managed discovery pipeline settings, then choose Recommended, which is the default and sets up the pipeline, or Not recommended, which skips it.

   Note

   If you don’t select Auto-discover services on my cluster, Instrumentation Hub can’t discover services in your Cluster and one-click instrumentation isn’t available.

Before you continue to the next step, review and tune the signals collected for Kubernetes monitoring. To customize the signals, open the Enable Kubernetes monitoring card’s menu and select one of the tuning options. Switch each signal on or off, then click Save.

Where each setting is managed

The tuning options fall into three groups, and each group is managed in a different place. Where a setting is managed determines whether you can change it later without redeploying:

• Logs and events settings are deployed and controlled through Grafana Cloud. You can change them later without a Helm upgrade in Instrumentation Hub, the setup and configuration tool under Connections in Grafana Cloud.
• Metrics settings are configured locally on your Cluster and aren’t managed through Grafana Cloud. To change them later, update your Helm values and redeploy the chart.
• Application observability settings change how the chart deploys collectors, and they appear in the install command the wizard generates. You can’t change them through Fleet Management, so make these choices before you copy the deployment command.

Metrics

To tune metrics, open the card’s menu and select Advanced metrics tuning. Metrics run directly on your Cluster. These settings are configured locally and aren’t managed through Grafana Cloud.

1. Switch these signals on or off, then click Save.

   • Cluster metrics: Collects CPU, memory, network, and storage metrics from Cluster Nodes and Pods.

     What is collectedWhy enable?

     Node resource usage; metrics about Pod health; Persistent volume usage; and Deployment, StatefulSet, and DaemonSet status.

     Essential for Cluster health monitoring; powers the Kubernetes Overview dashboard; tracks resource utilization and capacity planning; and detects Pod crashes, OOM kills, and so on.

   • Cost metrics (costMetrics): Collects resource costs per namespace and workload to identify expensive Pods and optimize spending. Uses OpenCost.

     What is collectedWhy enable?Private registry settings

     Resource cost attribution by namespace, workload, and Pod; CPU and memory cost breakdowns; Cloud provider pricing data integration; and cost efficiency metrics.

     Shows cost data; helps identify expensive workloads; enables FinOps and cost optimization; and tracks spending trends over time.

     If your environment pulls container images from a private registry, you must configure OpenCost’s image source after you deploy. Without this setting, OpenCost Pods become stuck in ImagePullBackOff and no cost metrics are collected. Refer to Configure OpenCost for a private image registry.

   • Energy metrics (hostMetrics.energyMetrics): Measures power consumption per Node and workload for carbon footprint tracking. Uses Kepler.

   • Pod and Service metrics by annotation: Scrapes metrics from Pods and Services that have the k8s.grafana.com/scrape annotation.

     What does this do?Why enable?Why disable?

     Automatically discovers running pods and services; dynamically scrapes Prometheus metrics from annotated Pods; detects new workloads without manual configuration; and uses Kubernetes annotations to find metrics endpoints.

     Enable for: zero-configuration metrics collection; automatically monitor new applications as they deploy; support microservices architectures with dynamic scaling; and find application-specific metrics beyond system metrics.

     Disable when: You want to explicitly define every scrape target; you’re concerned about discovering unintended metrics endpoints; your cluster has very strict network policies.

   • Prometheus Operator objects: Adds scrape targets from ServiceMonitor, PodMonitor, and Probe objects.

     What does this do?Why enable?Why disable?

     Discovers and monitors Prometheus Operator CRDs: ServiceMonitor, which defines how to scrape metrics from Kubernetes services; PodMonitor, which defines how to scrape metrics from Pods; and Probe, which defines blackbox probing of endpoints.

     Enable when: you’re already using Prometheus Operator in your Cluster; you have existing ServiceMonitor/PodMonitor definitions; you want to use your existing Prometheus configurations; you want to enable migration from Prometheus Operator to Grafana Alloy.

     Disable when: You’re not using Prometheus Operator Objects; you prefer using the native configuration of Grafana Alloy.

Logs and Events

To tune logs and events, open the card’s menu and select Advanced logs and events tuning. These options are deployed and controlled through Grafana Cloud, so you can update and scale instrumentation without changing your Cluster configuration.

1. Switch these signals on or off, then click Save.
   • Cluster events: Collects Kubernetes events, including Pod failures, scheduling events, and resource state changes.
     What is collectedWhy enable?

     Generated from: the scheduler (assigning Pods to Nodes); kubelet (managing Pods on Nodes); the controller manager (handling scaling, deployments, and so on); the API server (processing requests).

     Troubleshooting scheduling & deployment issues; see why a Pod isn’t starting (no Nodes with enough memory); tracking resource lifecycle changes (Pod created → scheduled → pulled → started → ready → terminated); detecting transient or recurring failures (repeated image pull errors, failed probes, or node taints); auditing Cluster activity (to identify which controller or user triggered changes).
   • Node logs: Collects kubelet and system logs from worker Nodes running systemd, including journald logs.
     What is collectedWhy enable?

     Logs from: the operating system (kernel logs, systemd, network drivers); Kubernetes node agents (like kubelet); container runtimes (Docker, containerd); system daemons (journald, syslog).

     Diagnosing node instability (memory exhaustion, CPU throttling, or disk space issues); debugging scheduling or startup failures (when Pods can’t start, the issue may be at the Node level); investigating network or storage problems; determining driver or volume mount failures; auditing system changes (Node reboots, kubelet restarts, or OS updates).
   • Pod logs: Collects application logs from running Pods. Available as podLogsViaLoki (Loki pipeline) or podLogsViaOpenTelemetry (native OTLP format, the OpenTelemetry protocol) in the Helm chart values.
     What is collectedWhy enable?

     Standard output (stdout) and standard error (stderr) Pod logs from the processes in the container, such as initialization messages, API request logs, warnings or errors, and application-specific information.

     Enable for: debugging issues, such as when an application crashes or behaves unexpectedly, logs reveal what went wrong; monitoring behavior, such as tracking normal operational messages (startup confirmation, API requests, or job completion); auditing events, to view logs that show what actions were taken by your app or scripts running inside containers; gaining performance insight, such as tracing slow operations or bottlenecks using timestamps and log levels.
   • Kubernetes manifests (kubernetesManifests): Collects Kubernetes resource manifests and changes as logs, using k8s-manifest-tail. Provides the data behind the View Manifest and Changes tabs on workloads and Pods.
     What is collectedWhy enable?

     Full resource manifests for Pods and workloads; creation, update, and deletion events for those resources; and the version history that powers the manifest diff viewer.

     When switched, you can compare a manifest against previous versions in a side-by-side diff; on Deployment detail page of k8s-manifest-tail, you can view a highlight on the graphs in the Optimization section to see if a deployment change correlates with a change in behavior.

Application observability

To configure application observability, open the card’s menu and select Advanced settings. These options change how the chart deploys collectors, and they appear in the install command. You can’t change them through Fleet Management.

1. Switch these signals on or off, then click Save.
   • OpenTelemetry receivers: Enables Alloy to receive metrics, logs, and traces from applications instrumented with OpenTelemetry.
     What is collectedWhat does this do?Why enable?Why disable?

     Metrics, logs, and traces from apps sending OTel data.

     Opens Alloy receiver ports to accept telemetry that your apps (or OTel SDK/collector) push to Grafana Cloud.

     Enable for: services using OTel SDKs so you can push traces/metrics to an endpoint; Application Observability (RED metrics, service map, trace correlation) in Grafana Cloud; existing Zipkin-instrumented apps when you want to receive traces from them; ensuring host-hours telemetry, which is required for Application Observability billing.

     Disable when: your apps have no instrumentation at all.
   • Application profiling: Captures CPU and memory profiles from Pods that have the profiles.grafana.com/scrape annotation.
     What does this do?Why enable?Why disable?

     Enables continuous profiling using eBPF, Java, or pprof; collects CPU flame graphs from running applications; captures function-level performance data; identifies code issues and performance bottlenecks.

     Enable when you want to: find expensive functions in your code; optimize application performance; debug CPU-intensive operations; identify memory allocation patterns.

     Disable when: You don’t need code-level profiling; you’re concerned about profiling overhead (~1-5% CPU); your applications are already well-optimized.
   • Zero-code instrumentation: Captures HTTP, gRPC, and database calls with no code changes, using eBPF and Grafana Beyla.

     Caution

     If you enable instrumentation with Beyla, this may affect your billing due to additional telemetry ingestion.

     What does this do?Why enable?Why disable?

     Enables instrumentation with Grafana Beyla, which uses eBPF to capture HTTP, gRPC, and database calls from applications running on your Cluster, with no code changes or instrumentation libraries required.

     Enable when you want to: See how Pod resource constraints affect application latency; correlate OOM kills with specific requests; understand resource consumption per endpoint; and connect infrastructure issues to user impact.

     Disable when: You only need basic infrastructure monitoring.
   • Forward traces to application receivers: Forwards traces captured by Beyla to the application receiver OTLP endpoint. Available only when both OpenTelemetry receivers and zero-code instrumentation are enabled.

After you finish tuning signals, click Next.

Deployment

1. In the Cluster name box, enter a unique, human-readable name for this Cluster. Use only lowercase letters, numbers, and hyphens, start with a letter, and end with a letter or number (up to 63 characters). This name appears in dashboards, alerts, and Cluster selectors.

2. In the Namespace box, enter the namespace where you want to install Grafana Alloy.

3. Optionally, under Custom labels, add key and value labels to help you search for, identify, and deploy pipelines in Fleet Management. Click Add more to add another label.

4. If you don’t already have the Grafana Helm charts repository, add it:

   Bash Copy

   helm repo add grafana https://grafana.github.io/helm-charts

   If you already have it, update it:

   Bash Copy

   helm repo update

5. Under Deployment code to copy, copy the generated helm upgrade --install command and run it in your terminal. Make sure your kubectl context points to the Cluster you want to monitor.

Verify your installation

Confirm that the collectors are running and data is arriving in Grafana Cloud:

1. Click Test connection to verify that the collectors install and send data to Grafana Cloud.

2. Click See cluster status to view the status of data collection. Data appears as the collectors start scraping and sending it to Grafana Cloud. This view shows the health of the different sources of metrics, Pod logs, and Cluster events, as well as any applicable version numbers.

This completes the configuration wizard.

Configure application instrumentation

If you chose to forward traces to application receivers, a list of endpoints appears. In your application that generates metrics, logs, or traces, enter the appropriate OTLP or Zipkin address.

Manage your installation

After you complete the wizard, use the following tasks to adjust, troubleshoot, and extend your installation.

Configure OpenCost for a private image registry

If you enabled cost metrics and your environment pulls container images from a private registry, configure OpenCost’s image source. Without this setting, OpenCost Pods become stuck in ImagePullBackOff and no cost metrics are collected.

1. Add this to your values.yaml file:

   YAML Copy

   telemetryServices:
     opencost:
       deploy: true
       opencost:
         exporter:
           image:
             registry: my.private.registry.example.com

2. Deploy or upgrade the chart:

   Bash Copy

   helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring \
     --values values.yaml

For other private registry options, refer to Private image registry.

Troubleshoot

Refer to Troubleshooting for any issues that occur after configuration.

Install any integrations

You can use Grafana integrations to monitor the health and status of services and applications running in your Kubernetes clusters.

To install a Kubernetes integration to begin scraping metrics:

1. From the main menu, navigate to Connections, and filter for Kubernetes.

2. Select the integration for the service you want to monitor.

3. Follow the instructions on the screen to copy and use the configuration snippet and install the integration.

4. After installing an integration, redeploy the configuration using the method you originally used.

Retrieve Helm values

If you installed Kubernetes Monitoring with the Helm CLI, you can retrieve the values for your configuration by using the helm get values command.