Customize the Kubernetes Monitoring Helm chart

Monitor infrastructure

Kubernetes Monitoring

Configure Kubernetes Monitoring

Configure with Helm chart

Customize Helm chart

Grafana Cloud

Customize the Kubernetes Monitoring Helm chart

After you complete the easy deployment process in Kubernetes Monitoring, you can further customize your configuration using the Helm chart. For example, you may want to collect more data, add more destinations, or need guidance on authentication settings.

The examples in the Kubernetes Monitoring Helm chart are complete examples designed to help you correctly alter your configuration by editing the values.yaml file. The following categories provide a means to find the example you need for customization.

After you make any customizations in values.yaml, redeploy the Helm chart to apply your changes.

Authentication

Use the following examples to configure authentication.

Example	Description
Bearer token	User a bearer token for a Prometheus, Loki, or OTLP destination.
Embedded secret	Embed the secret data directly into the destination configuration.
External secret	Use pre-existing secrets to authenticate to external services.
Oauth2	Use OAuth2 for authentication.
Sigv4	Configure a Prometheus destination using the AWS Signature Version 4 authentication method.

Data collection

Alloy

Gather metrics from Grafana Alloy.

Application data

The Application Observability feature encompasses receiving data via various receivers, processing that data, and delivering it to the specified destinations. This example shows the settings to collect telemetry data from an application.

Grafana

Gather metrics and logs from Grafana.

Grafana Loki

Gather metrics and logs from Grafana Loki.

Kubernetes infrastructure data

Example	Description
cert-manager	Gather metrics from cert-manager.
Cluster events	Gather Kubernetes Cluster events from the Kubernetes API server, and deliver them to a logs destination.
Cluster metrics	Gather metrics about the Kubernetes Cluster and deliver them to a metrics destination. This includes using services and tools such as Node Exporter, kube-state-metrics, `kubelet`, and cadvisor.
Cluster metrics with Istio Service Mesh	Gather metrics from Alloy clustering when Istio Service Mesh is enabled and has deployed the Istio sidecar to the Pods in the Cluster.
Cluster and control plane metrics	Gather metrics about the Kubernetes Cluster, including its control plane components, and deliver them to a metrics destination.
etcd	Gather metrics from etcd.
Log metrics	Generate metrics captured from Pod logs.
Node logs	Gather logs from the Nodes in your Kubernetes Cluster. This is useful when you create your own Kubernetes Cluster with `kubeadm`, because `kubelet` runs as a systemd service on Linux. This example shows gathering logs from the journald services.
Pod logs	Gather logs from the Pods in your Kubernetes Cluster.
Automatically discovered Pods and Services	Kubernetes Pods or Services are automatically discovered and scraped by the collector.
PodMonitors, ServiceMonitors, and Probes	Enable discovering PodMonitors, ServiceMonitors, and Probes in your Kubernetes Cluster and using them to scrape metrics.
Profiles	Gather profiles from your Kubernetes Cluster, and deliver them to Pyroscope.
Tolerations	Allow workloads to run on Nodes with specific taints or prevent workloads from running on Nodes with specific taints.

Monitoring databases that monitor metrics, logs, or traces

With meta monitoring, you can monitor these databases and send the data to Grafana Cloud:

Enterprise databases: Grafana Enterprise Metrics, Grafana Enterprise Logs, and Grafana Enterprise Traces
OSS databases: Mimir, Loki, and Tempo

MySQL

Gather metrics and logs from MySQL.

Destinations and proxies

Specify single or multiple destinations, whether a local service deployed on the same cluster, or a remote SaaS service. Use proxy URLs and TLS settings to send data to external services.

Example	Description
Loki	Send logs using the loki protocol to a logs destination.
OTLP endpoint	Send all your telemetry data to a single destination using OTLP destination.
OTLP or OTLPHTTP	Send metrics, logs, or traces using the OTLP protocol to an OTLP destination.
Proxies for external services	Use proxy URLs and TLS settings to send data to external services.
Prometheus	Send metrics using the remote write protocol to a metrics destination.
Pyroscope	Send metrics using the pyroscope protocol to a profiles destination.

Discovery

You can customize data collection during the discovery phase.

Example	Description
Annotations	Use annotations to automatically discovered and gather metrics from Kubernetes Pods and Services. You can use these annotations to further customize by job, instance, path, port number or name, scheme, and scrape interval.
Automatic discovery using Prometheus annotations	Use Prometheus-style annotations to enable Alloy to discovery metrics, and customize the path and port number.
Extra discovery rules and labels	You can refine what services are discovered and control the target’s label using these extra rules and labels.
MongoDB Atlas databases	Gather metrics from MongoDB Atlas databases with Alloy using the discovery.http component.
MySQL	Gather metrics and logs from MySQL.
Namespace exclusion	Define what namespaces to exclude from data discovery and include all other namespaces.
Namespace inclusion	Define what namespaces to include in data discovery and exclude all other namespaces.
Pod labels and annotations	Include labels and annotations set on the Kubernetes Pod to the telemetry data for that Pod.

For more information on Grafana Alloy labels and relabeling, refer to:

Alloy discovery.relabel to control metrics collection or standardize target labels
Alloy __meta labels to refine data collection of Kubernetes resources. An example of this is shown in etcd.

For examples of rules and labeling after the discovery phase, refer to Processing and labeling.

Helm chart deployment

Deploy the Kubernetes Monitoring Helm chart using Terraform.

Instrumentation for applications

Automatically instrument your applications for telemetry collection.

Example	Description
Metrics	Deploy Grafana Beyla to automatically instrument your application for metrics collection.
Metrics and traces	Deploy Grafana Beyla to automatically instrument your application for metrics collection, and generate traces for your application.

Platforms

Customize your platform to work correctly with Kubernetes Monitoring.

Example	Description
Azure AKS	Enable Kubernetes Monitoring to work correctly with Azure AKS Clusters.
EKS Fargate	Gather Pod logs on an EKS Fargate Cluster so that Kubernetes Monitoring works correctly.
GKE Autopilot	Enable Kubernetes Monitoring to work correctly on GKE Autopilot Clusters.
OpenShift	Enable Kubernetes Monitoring to work correctly on OpenShift Clusters.

Processing and labeling

After data collection during the processing phase, you can enable additional processing for telemetry data, refine the metrics you want to keep or drop, and change existing labels or add labels.

Example	Description
Additional labels	Use `extraDiscoveryRules` to further refine data collection.
Additional processing	Enable additional processing for logs and metrics, such as `extraMetricProcessingRules` and `extraLogProcessingStages`.
Metrics tuning	Include or exclude metrics to be sent to a metrics destination by using a rule to exclude or include metrics. The default ConfigMap that results from the configuration process creates allowlists. These allowlists are configured to keep a subset of metrics used by Kubernetes Monitoring. With metrics tuning, you can add or exclude any metrics from the default allowlist.

To learn more about labels and relabeling, refer to:

To learn more about metrics tuning and allowlists, refer to:

For examples of labels and annotations during the discovery phase, refer to Discovery.

Private image registry

To support environments that are air-gapped or should be excluded from using public image registries, you can do either of the following:

Globally override the container image registries for every subchart by using a global object.
Override the individual container image registries for every subchart by using these values.

Remote configuration

Enable Grafana Alloy to fetch and load the configuration from a remote endpoint.

Scaling and reliability

Enhance scaling and reliability where needed.

Example	Description
Alloy auto scaling	Enable an Alloy instance to scale up and down based on CPU and memory use.
Collector storage	Enable metric scraping to use a Write-Ahead Log (WAL) to store metrics in case of a scrape failure. Enable log gathering to use a volume to store log file positions. This provides a starting point for reading logs after a restart.
Sharded kube-state-metrics	Shard kube-state-metrics to improve scaling.