Documentation Grafana Cloud Observe and act Monitor infrastructure Kubernetes Monitoring Configure Kubernetes Monitoring Configure with Helm chart Configuration steps

Configuration steps for Kubernetes Monitoring with Helm chart

Complete the configuration process on the Cluster configuration tab. The wizard guides you through five steps: Backend and distribution, Access token, Choose setup, Monitoring, and Deployment. Complete each step, then click Next.

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on this Configuration page.

Try it

Before you begin

Make sure you have met the prerequisites required for these configuration steps.

To deploy Kubernetes Monitoring with the Helm chart, you need:

• A Grafana Cloud account
• The Admin role to install alerts
• To have completed the activation of the app
• A Kubernetes Cluster, environment, or fleet you want to monitor
• Helm installed
• The kubectl and Helm command-line tools

• Appropriate versions of items related to:
  • kube-state-metrics: Uses client-go to communicate with Clusters. For Kubernetes client-go version compatibility and any other related details, refer to kube-state-metrics.
  • OpenCost: Requires Kubernetes 1.8+ clusters.
  • Visualizations for storage: Require Helm chart release v1.5.1 or later

Backend and distribution

Install the required backend rules and choose the platform for your Cluster.

1. Under Backend Installation, click Install to install the required, preconfigured alert rules and recording rules to your Grafana instance. When they’re installed, the status shows Alert rules and recording rules installed.

2. Under Select your Kubernetes platform, choose your platform:

   • Kubernetes, for EKS on EC2, GKE, and similar distributions
   • Azure AKS
   • GKE Autopilot
   • EKS on Fargate
   • IBM Cloud
   • OpenShift

3. Click Next.

Access token

An access policy token is required for Grafana Alloy to send metrics and logs to Grafana Cloud. Choose how the Helm chart authenticates to Grafana Cloud. You can create a new access policy token, use an existing token, or point the chart at a stored Kubernetes Secret that already exists in your Cluster. Refer to Grafana Cloud Access Policies for more information.

To create a new token:

1. Click Create a new token.

2. In the Access policy token name box, enter a name. This name helps you view and revoke the token in Grafana.com later.

3. In the Expiration date box, select or enter an expiration in days, or keep No expiry.

   The token uses the predefined scopes metrics:read and set:alloy-data-write.

4. Click Create token.

To use an existing token:

1. Click Use an existing token.

2. Paste your token into the box.

To use a stored Kubernetes Secret:

1. Click Use a stored Kubernetes Secret.

2. In the Secret name box, enter the name of the Secret that already exists in your Cluster.

3. In the Namespace box, keep the installation namespace or enter the namespace that holds the Secret.

4. Optionally, expand the advanced options to change the Username key and Password key if your Secret doesn’t use the default keys username and password.

   The Secret must hold your Grafana Cloud instance ID as the username and an access policy token as the password, with the metrics:read and set:alloy-data-write scopes. The configuration page provides a kubectl command, prefilled with your instance ID, that you can copy to create the Secret. Because the chart reads the token from the Secret at runtime, your token isn’t stored in Grafana and doesn’t appear in the generated Helm command.

5. Click Next.

Choose setup

Select a preconfigured Alloy deployment based on the level of observability you need. Each option determines what telemetry you can collect now and what you can enable later without reinstalling. Expand Learn what this deployment includes on any option to see its details. Your choice sets the collector presets in the generated Helm values.

• Scalable - High performance (Recommended): Runs one Alloy collector on every Node (a DaemonSet) plus a separate clustered backend (a Deployment) for cluster-wide aggregation. The per-Node collectors use the xlarge size and vertically autoscale between 1 and 4 CPU cores and 2 to 8 GiB of memory. The backend horizontally autoscales from 1 to 5 replicas when CPU passes 75% or memory passes 80%, and it buffers to persistent storage (PVC). Best for large-scale production environments, high-traffic systems, and mission-critical services.
• Standard - Balanced cluster setup: Runs one Alloy collector on every Node (a DaemonSet) plus a clustered backend (a Deployment) for aggregation, both at the fixed large size (0.5 to 2 CPU cores and 1 to 2 GiB of memory per instance). It doesn’t autoscale and has limited buffering. Best for most production workloads, growing teams, and medium to large Clusters.
• Lightweight - Quick start: Runs a single Alloy collector on every Node (a DaemonSet) at the large size, with no separate backend, no cluster-wide aggregation, and no persistent storage. Best for smaller Clusters, early-stage environments, and cost-conscious setups.

Your tier selection also seeds the default monitoring options on the Monitoring step, which you can still change. Scalable and Standard enable the recommended signals, which is everything except Node logs and trace forwarding. Lightweight enables only Cluster metrics and Cluster events.

Click Next.

Monitoring

On the Remote monitoring and service discovery step, enable the remote monitoring features and managed pipelines you want from Grafana Cloud. You can change these selections later in Instrumentation Hub without a Helm upgrade. Kubernetes monitoring is required by this flow, and auto-discovery of services is optional.

1. Select Enable Kubernetes monitoring to instrument your Cluster with the recommended signals for most production Clusters. You can customize the signals later in Instrumentation Hub.

   Note

   After you activate Kubernetes monitoring, billing begins based on your host and container hours.

2. Optionally, select Auto-discover services on my cluster to create a managed discovery pipeline in Fleet Management. No application telemetry is collected unless you enable instrumentation. To change this, open the card’s menu, select Managed discovery pipeline settings, then choose Recommended or Not recommended.

3. Click Next.

To customize the signals collected for Kubernetes monitoring, open the Enable Kubernetes monitoring card’s menu and select one of the tuning options. Switch each signal on or off, then click Save.

Metrics

To tune metrics, open the card’s menu and select Advanced metrics tuning. Metrics run directly on your Cluster. These settings are configured locally and aren’t managed through Grafana Cloud.

1. Switch these signals on or off, then click Save.

   • Cluster metrics: Collects CPU, memory, network, and storage metrics from Cluster Nodes and Pods.

     What is collectedWhy enable?

     Node resource usage; metrics about Pod health; Persistent volume usage; and Deployment, StatefulSet, and DaemonSet status.

     Essential for Cluster health monitoring; powers the Kubernetes Overview dashboard; tracks resource utilization and capacity planning; and detects Pod crashes, OOM kills, and so on.

   • Cost metrics (costMetrics): Collects resource costs per namespace and workload to identify expensive Pods and optimize spending. Uses OpenCost.

     What is collectedWhy enable?Private registry settings

     Resource cost attribution by namespace, workload, and Pod; CPU and memory cost breakdowns; Cloud provider pricing data integration; and cost efficiency metrics.

     Shows cost data; helps identify expensive workloads; enables FinOps and cost optimization; and tracks spending trends over time.

     To configure OpenCost’s image source in a private registry environment, complete the following steps. Without this setting, OpenCost pods become stuck in ImagePullBackOff and no cost metrics will be collected.

     1. Add this to your values.yaml file:

        YAML Copy

        telemetryServices:
          opencost:
            deploy: true
            opencost:
              exporter:
                image:
                  registry: my.private.registry.example.com

     2. Then deploy or upgrade:

        Bash Copy

        helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring \
          --values values.yaml

   • Energy metrics (hostMetrics.energyMetrics): Measures power consumption per Node and workload for carbon footprint tracking. Uses Kepler.

   • Pod and Service metrics by annotation: Scrapes metrics from Pods and Services that have the k8s.grafana.com/scrape annotation.

     What does this do?Why enable?Why disable?

     Automatically discovers running pods and services; dynamically scrapes Prometheus metrics from annotated Pods; detects new workloads without manual configuration; and uses Kubernetes annotations to find metrics endpoints.

     Enable for: zero-configuration metrics collection; automatically monitor new applications as they deploy; support microservices architectures with dynamic scaling; and find application-specific metrics beyond system metrics.

     Disable when: You want to explicitly define every scrape target; you’re concerned about discovering unintended metrics endpoints; your cluster has very strict network policies.

   • Prometheus Operator objects: Adds scrape targets from ServiceMonitor, PodMonitor, and Probe objects.

     What does this do?Why enable?Why disable?

     Discovers and monitors Prometheus Operator CRDs: ServiceMonitor, which defines how to scrape metrics from Kubernetes services; PodMonitor, which defines how to scrape metrics from Pods; and Probe, which defines blackbox probing of endpoints.

     Enable when: you’re already using Prometheus Operator in your Cluster; you have existing ServiceMonitor/PodMonitor definitions; you want to leverage existing Prometheus configurations; you want to enable migration from Prometheus Operator to Grafana Alloy.

     Disable when: You’re not using Prometheus Operator Objects; you prefer using the native configuration of Grafana Alloy.

Logs and Events

To tune logs and events, open the card’s menu and select Advanced logs and events tuning. These options are deployed and controlled through Grafana Cloud, so you can update and scale instrumentation without changing your Cluster configuration.

1. Switch these signals on or off, then click Save.
   • Cluster events: Collects Kubernetes events, including Pod failures, scheduling events, and resource state changes.
     What is collectedWhy enable?

     Generated from: the scheduler (assigning Pods to Nodes); kubelet (managing Pods on Nodes); the controller manager (handling scaling, deployments, and so on); the API server (processing requests).

     Troubleshooting scheduling & deployment issues; see why a Pod isn’t starting (no Nodes with enough memory); tracking resource lifecycle changes (Pod created → scheduled → pulled → started → ready → terminated); detecting transient or recurring failures (repeated image pull errors, failed probes, or node taints); auditing Cluster activity (to identify which controller or user triggered changes).
   • Node logs: Collects kubelet and system logs from worker Nodes running systemd, including journald logs.
     What is collectedWhy enable?

     Logs from: the operating system (kernel logs, systemd, network drivers); Kubernetes node agents (like kubelet); container runtimes (Docker, containerd); system daemons (journald, syslog).

     Diagnosing node instability (memory exhaustion, CPU throttling, or disk space issues); debugging scheduling or startup failures (when Pods can’t start, the issue may be at the Node level); investigating network or storage problems; determining driver or volume mount failures; auditing system changes (Node reboots, kubelet restarts, or OS updates).
   • Pod logs: Collects application logs from running Pods. Available as podLogsViaLoki (Loki pipeline) or podLogsViaOpenTelemetry (native OTLP format) in the Helm chart values.
     What is collectedWhy enable?

     Standard output (stdout) and standard error (stderr) Pod logs from the processes in the container, such as initialization messages, API request logs, warnings or errors, and application-specific information.

     Enable for: debugging issues, such as when an application crashes or behaves unexpectedly, logs reveal what went wrong; monitoring behavior, such as tracking normal operational messages (startup confirmation, API requests, or job completion); auditing events, to view logs that show what actions were taken by your app or scripts running inside containers; gaining performance insight, such as tracing slow operations or bottlenecks using timestamps and log levels.
   • Kubernetes manifests (kubernetesManifests): Collects Kubernetes resource manifests and changes as logs, using k8s-manifest-tail. Powers the View Manifest and Changes tabs on workloads and Pods.
     What is collectedWhy enable?

     Full resource manifests for Pods and workloads; creation, update, and deletion events for those resources; and the version history that powers the manifest diff viewer.

     When switched, you can compare a manifest against previous versions in a side-by-side diff; on Deployment detail page of k8s-manifest-tail, you can view a highlight on the graphs in the Optimization section to see if a deployment change correlates with a change in behavior.

Application observability

To configure application observability, open the card’s menu and select Advanced settings. These options change the Helm chart’s collector shape and are emitted into the install command. They aren’t configurable through Fleet Management.

1. Switch these signals on or off, then click Save.
   • OpenTelemetry receivers: Enables Alloy to receive metrics, logs, and traces from applications instrumented with OpenTelemetry.
     What is collectedWhat does this do?Why enable?Why disable?

     Metrics, logs, and traces from apps sending OTel data.

     Opens Alloy receiver ports to accept telemetry that your apps (or OTel SDK/collector) push to Grafana Cloud.

     Enable for: services using OTel SDKs so you can push traces/metrics to an endpoint; Application Observability (RED metrics, service map, trace correlation) in Grafana Cloud; existing Zipkin-instrumented apps when you want to receive traces from them; ensuring host-hours telemetry, which is required for Application Observability billing.

     Disable when: your apps have no instrumentation at all.
   • Application profiling: Captures CPU and memory profiles from Pods that have the profiles.grafana.com/scrape annotation.
     What does this do?Why enable?Why disable?

     Enables continuous profiling using eBPF, Java, or pprof; collects CPU flame graphs from running applications; captures function-level performance data; identifies code issues and performance bottlenecks.

     Enable when you want to: find expensive functions in your code; optimize application performance; debug CPU-intensive operations; identify memory allocation patterns.

     Disable when: You don’t need code-level profiling; you’re concerned about profiling overhead (~1-5% CPU); your applications are already well-optimized.
   • Zero-code instrumentation: Captures HTTP, gRPC, and database calls with no code changes, using eBPF and Grafana Beyla.

     Caution

     If you enable instrumentation with Beyla, this may affect your billing due to additional telemetry ingestion.

     What does this do?Why enable?Why disable?

     Correlates Pod metrics with application traces; links infrastructure metrics to application performance; enables unified views of resource usage and request patterns; powers the correlation features in Pod detail views.

     Enable when you want to: See how Pod resource constraints affect application latency; correlate OOM kills with specific requests; understand resource consumption per endpoint; and connect infrastructure issues to user impact.

     Disable when: You only need basic infrastructure monitoring.
   • Forward traces to application receivers: Forwards traces captured by Beyla to the application receiver OTLP endpoint. Available only when both OpenTelemetry receivers and zero-code instrumentation are enabled.

Deployment

1. In the Cluster name box, enter a unique, human-readable name for this Cluster. Use only lowercase letters, numbers, and hyphens, start with a letter, and end with a letter or number (up to 63 characters). This name appears in dashboards, alerts, and Cluster selectors.

2. In the Namespace box, enter the namespace where you want to install Grafana Alloy.

3. Optionally, under Custom labels, add key and value labels to help you search for, identify, and deploy pipelines in Fleet Management. Click Add more to add another label.

4. If you don’t already have the Grafana Helm charts repository, add it:

   Bash Copy

   helm repo add grafana https://grafana.github.io/helm-charts

   If you already have it, update it:

   Bash Copy

   helm repo update

5. Under Deployment code to copy, copy the generated helm upgrade --install command and run it in your terminal. Make sure your kubectl context points to the Cluster you want to monitor.

6. Click Test connection to verify that the collectors install and send data to Grafana Cloud.

Configure application instrumentation

If you chose to forward traces to application receivers, a list of endpoints appear. In your application that generates metrics, logs, or traces, enter the appropriate OTLP or Zipkin address.

Click See cluster status to view the status of data collection. Your data becomes populated as the system components begin scraping and sending data to Grafana Cloud. This view shows the health of the different sources of metrics, Pod logs, and Cluster events, as well as any applicable version numbers.

Troubleshoot

Refer to Troubleshooting for any issues that occur after configuration.

Install any integrations

You can use Grafana integrations to monitor the health and status of services and applications running in your Kubernetes clusters.

To install a Kubernetes integration to begin scraping metrics:

1. From the main menu, navigate to Connections, and filter for Kubernetes.

2. Select the integration for the service you want to monitor.

3. Follow the instructions on the screen to copy and use the configuration snippet and install the integration.

4. After installing an integration, redeploy the configuration using the method you originally used.

Retrieve Helm values

If you installed Kubernetes Monitoring with the Helm CLI, you can retrieve the values for your configuration by using the helm get values command.