--- title: "Customize the Kubernetes Monitoring Helm chart | Grafana Cloud documentation" description: "How to customize the Helm chart for Kubernetes Monitoring" --- # Customize the Kubernetes Monitoring Helm chart After you complete the deployment process of the Kubernetes Monitoring Helm chart, you can further customize your configuration. For example, you may want to collect more [data](#data-collection), add more [destinations](#destinations-and-proxies), or need guidance on [authentication](#authentication) settings. The [examples in the Kubernetes Monitoring Helm chart](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples) are complete examples designed to help you correctly alter your configuration by editing the [values.yaml file](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/values.yaml). The following categories provide a means to find the example you need for customization. After you make any customizations in values.yaml, redeploy the Helm chart to apply your changes. ## Helm chart version Each Kubernetes Monitoring Helm chart version has added functionality. To take advantage of features of an updated version: - Check the [Helm chart release notes](https://github.com/grafana/k8s-monitoring-helm/releases) for the updates available in each version. - Install the latest version of the Helm chart by running: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring -f values.yaml ``` If you need to migrate from an earlier version, refer to the steps in [Version migration](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/migrate-helm-chart/). ## Common inclusion or exclusion filters The following tables show common ways to include or exclude sources from being collected from various sources. ### Metric sources from Kubernetes Cluster Expand table | Source | YAML | Filters | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|--------------------------------------| | [Cluster metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-cluster-metrics/README.md) | `clusterMetrics` | Source, metric, namespace | | [Host metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-host-metrics/README.md) | `hostMetrics` | Source, metric, namespace | | [Energy metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-host-metrics/README.md) | `hostMetrics.energyMetrics` | Source, metric, namespace | | [Cost metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-cost-metrics/README.md) | `costMetrics` | Source, metric, namespace | | [Auto discovery with annotations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-annotation-autodiscovery/README.md) | `annotationAutodiscovery` | Metric, label, annotation, namespace | | [Prometheus Operator objects](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-prometheus-operator-objects/README.md) | `prometheusOperatorObjects` | Metric, label, namespace | ### Log sources from Kubernetes Cluster Expand table | Source | YAML | Filters | |------------------------------------------------------------------------------------------------------------------------------------------|-----------------|--------------------------| | [Cluster events](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-cluster-events/README.md) | `clusterEvents` | Level, reason, namespace | | [Node logs](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/charts/feature-node-logs) | `nodeLogs` | Unit | ### Pod logs from Kubernetes Cluster Expand table | Source | YAML | Filters | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|----------------| | [Pod logs via Loki](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-pod-logs-via-loki/README.md) | `podLogsViaLoki` | Pod, namespace | | [Pod logs via OpenTelemetry](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-pod-logs-via-opentelemetry/README.md) | `podLogsViaOpenTelemetry` | Pod, namespace | | [Pod logs objects](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-pod-logs-objects/README.md) | `podLogsObjects` | Pod, namespace | | [Pod logs via Kubernetes API](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-pod-logs-via-kubernetes-api/README.md) | `podLogsViaKubernetesApi` | Pod, namespace | ### Application telemetry Expand table | Source | YAML | Filters | |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------|------------------------------| | [Application receivers](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-application-observability/README.md) | `applicationObservability` | Metric, log, trace | | [Metrics and traces of inbound and outbound calls](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-auto-instrumentation/README.md) | `autoInstrumentation` | Label, service | | [CPU profiling](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-profiling/README.md) | `profiling` | Annotation, label, namespace | | [Profiles receiver](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-profiles-receiver/README.md) | `profilesReceiver` | Namespace | ## Authentication Use the following examples to configure authentication. Expand table | Example | Description | |-------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| | [Bearer token](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/auth/bearer-token/README.md) | User a bearer token for a Prometheus, Loki, or OTLP destination. | | [Embedded secret](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/auth/embedded-secrets/README.md) | Embed the secret data directly into the destination configuration. | | [External secret](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/auth/external-secrets/README.md) | Use pre-existing secrets to authenticate to external services. | | [Oauth2](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/auth/oauth2/README.md) | Use OAuth2 for authentication. | | [Sigv4](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/auth/sigv4/README.md) | Configure a Prometheus destination using the AWS Signature Version 4 authentication method. | ## Data collection ### Alloy Gather metrics from [Grafana Alloy](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/alloy/README.md). ### Application data The Application Observability feature encompasses receiving data via various receivers, processing that data, and delivering it to the specified destinations. [This example](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/application-observability/default/README.md) shows the settings to collect telemetry data from an application. ### Grafana Gather metrics and logs from [Grafana](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-integrations/docs/integrations/grafana.md). ### Grafana Loki Gather metrics and logs from [Grafana Loki](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/loki/README.md). ### Kubernetes infrastructure data Expand table | Example | Description | |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [cert-manager](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/cert-manager/README.md) | Gather metrics from [cert-manager](https://cert-manager.io/). | | [Cluster events](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/cluster-events/default/README.md) | Gather Kubernetes Cluster events from the Kubernetes API server, and deliver them to a logs destination. | | [Cluster metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/cluster-metrics/default/README.md) | Gather metrics about the Kubernetes Cluster and deliver them to a metrics destination. This includes using services and tools such as [Node Exporter](https://github.com/prometheus/node_exporter), [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics), [`kubelet`](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#:~:text=The%20kubelet%20is%20the%20primary,object%20that%20describes%20a%20pod.), and [cadvisor](https://github.com/google/cadvisor). | | [Cluster metrics with Istio Service Mesh](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/istio-service-mesh/README.md) | Gather metrics from Alloy clustering when Istio Service Mesh is enabled and has deployed the Istio sidecar to the Pods in the Cluster. | | [Cluster and control plane metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/cluster-metrics/control-plane-monitoring/README.md) | Gather metrics about the Kubernetes Cluster, including its control plane components, and deliver them to a metrics destination. | | [etcd](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/etcd/README.md) | Gather metrics from [etcd](https://etcd.io/). | | [Log metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/log-metrics/README.md) | Generate metrics captured from Pod logs. | | [Node logs](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/node-logs/default/README.md) | Gather logs from the Nodes in your Kubernetes Cluster. This is useful when you create your own Kubernetes Cluster with `kubeadm`, because `kubelet` runs as a systemd service on Linux. This example shows gathering logs from the journald services. | | [Pod logs via Loki](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/pod-logs-via-loki/default/README.md) | Gather logs from the Pods in your Kubernetes Cluster using the Loki pipeline. | | [Pod logs via OpenTelemetry](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/pod-logs-via-opentelemetry/README.md) | Gather logs from the Pods in your Kubernetes Cluster natively in OTLP format. | | [Pod logs objects](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/pod-logs-objects/default/README.md) | Gather logs from the Pods in your Kubernetes Cluster using PodLogs objects. | | [Pod logs via Kubernetes API](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/pod-logs-via-kubernetes-api/default/README.md) | Gather Pod logs by streaming them from the Kubernetes API. Useful when you cannot use the HostPath volume mount method. | | [Automatically discovered Pods and Services](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/annotation-autodiscovery/default/README.md) | Kubernetes Pods or Services are automatically discovered and scraped by the collector. | | [PodMonitors, ServiceMonitors, and Probes](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/prometheus-operator-objects/default/README.md) | Enable discovering PodMonitors, ServiceMonitors, and Probes in your Kubernetes Cluster and using them to scrape metrics. | | [Profiles](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/profiling/default/README.md) | Gather profiles from your Kubernetes Cluster, and deliver them to Pyroscope. | | [Profiles receiver](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/profiles-receiver/default/README.md) | Receive profiles from applications and deliver them to Pyroscope. | | [Database Observability for MySQL](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/database-observability/mysql/README.md) | Enable database observability for MySQL databases. | | [Database Observability for PostgreSQL](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/database-observability/postgresql/README.md) | Enable database observability for PostgreSQL databases. | | [Tolerations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/tolerations/README.md) | Allow workloads to run on Nodes with specific taints or prevent workloads from running on Nodes with specific taints. | ### Monitoring databases that monitor metrics, logs, or traces With [meta monitoring](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/meta-monitoring/README.md), you can monitor these databases and send the data to Grafana Cloud: - Enterprise databases: [Grafana Enterprise Metrics](/docs/enterprise-metrics/latest/), [Grafana Enterprise Logs](/docs/enterprise-logs/latest/), and [Grafana Enterprise Traces](/docs/enterprise-traces/latest/) - OSS databases: [Mimir](/docs/mimir/latest/), [Loki](/docs/loki/latest/), and [Tempo](/docs/tempo/latest/) ### DCGM Exporter Gather GPU metrics from [DCGM Exporter](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/dcgm-exporter/README.md). ### Grafana Mimir Gather metrics and logs from [Grafana Mimir](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/mimir/README.md). ### Multiple integrations Deploy [multiple service integrations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/multiple/README.md) alongside one another. ### MySQL Gather metrics and logs from [MySQL](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/features/integrations/mysql). ### PostgreSQL Gather metrics and logs from [PostgreSQL](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/postgresql/README.md). ## Destinations and proxies Specify single or multiple destinations, whether a local service deployed on the same cluster, or a remote SaaS service. Use proxy URLs and TLS settings to send data to external services. Expand table | Example | Description | |----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| | [Loki](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/destinations/README.md) | Send logs using the loki protocol to a logs destination. | | [OTLP endpoint](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/destinations/otlp-endpoint/README.md) | Send all your telemetry data to a single destination using [OTLP destination](/docs/grafana-cloud/send-data/otlp/send-data-otlp/). | | [OTLP or OTLPHTTP](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/destinations/README.md) | Send metrics, logs, or traces using the OTLP protocol to an OTLP destination. | | [Proxies for external services](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/proxies/README.md) | Use proxy URLs and TLS settings to send data to external services. | | [Prometheus](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/destinations/README.md) | Send metrics using the remote write protocol to a metrics destination. | | [Pyroscope](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/destinations/README.md) | Send metrics using the pyroscope protocol to a profiles destination. | ## Discovery You can customize data collection during the discovery phase. Expand table | Example | Description | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Annotations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-annotation-autodiscovery/README.md) | Use annotations to automatically discovered and gather metrics from Kubernetes Pods and Services. You can use these annotations to further customize by job, instance, path, port number or name, scheme, and scrape interval. Also refer to [Kubernetes annotations](#kubernetes-annotations) for more information. | | [Automatic discovery using Prometheus annotations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/annotation-autodiscovery/prom-annotations/README.md) | Use Prometheus-style annotations to enable Alloy to discovery metrics, and customize the path and port number. | | [Extra discovery rules and labels](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/extra-rules/README.md) | You can refine what services are discovered and control the target’s label using these extra rules and labels. | | [MongoDB Atlas databases](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/service-integrations/mongodb-atlas/README.md) | Gather metrics from MongoDB Atlas databases with Alloy using the discovery.http component. | | [MySQL](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/integrations/mysql/README.md) | Gather metrics and logs from MySQL. | | [Namespace exclusion](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/exclude-by-namespace/README.md) | Define what namespaces to exclude from data discovery and include all other namespaces. | | [Namespace inclusion](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/include-by-namespace/README.md) | Define what namespaces to include in data discovery and exclude all other namespaces. | | [Namespace labels and annotations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/namespace-labels-and-annotations/README.md) | Include labels and annotations set on the Kubernetes namespace to the telemetry data for resources in that namespace. | | [Node labels](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/node-labels/README.md) | Include labels set on the Kubernetes Node to the telemetry data for that Node. | | [Pod labels and annotations](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/pod-labels-and-annotations/README.md) | Include labels and annotations set on the Kubernetes Pod to the telemetry data for that Pod. | For more information on Grafana Alloy labels and relabeling, refer to: - [Alloy `discovery.relabel`](/docs/grafana-cloud/send-data/alloy/reference/components/discovery/discovery.relabel/) to control metrics collection or standardize target labels - [Alloy `__meta` labels](/docs/grafana-cloud/send-data/alloy/reference/components/discovery/discovery.kubernetes/#node-role) to refine data collection of Kubernetes resources. An example of this is shown in [etcd](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/alloyModules/modules/databases/kv/etcd/metrics.alloy). For examples of rules and labeling *after* the discovery phase, refer to [Processing and labeling](#processing-and-labeling). ## Helm chart deployment Deploy the Kubernetes Monitoring Helm chart using alternative methods or customize deployment settings. Expand table | Example | Description | |------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------| | [Argo CD](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/deployment-alternatives/argocd/README.md) | Deploy using Argo CD. | | [Name overrides](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/name-overrides/fullname-overrides/README.md) | Override resource names in the Helm chart. | | [RBAC without Cluster Roles](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/rbac-no-cluster-roles/README.md) | Deploy without ClusterRoles or ClusterRoleBindings. | | [Terraform](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/deployment-alternatives/terraform/README.md) | Deploy using Terraform. | ## Instrumentation for applications Automatically instrument your applications for telemetry collection. Expand table | Example | Description | |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| | [Metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/auto-instrumentation/beyla-metrics/README.md) | Deploy Grafana [Beyla](/docs/beyla/latest/) to instrument your application using zero code for metrics collection. | | [Metrics and traces](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/features/auto-instrumentation/beyla-metrics-and-traces/README.md) | Deploy Grafana Beyla to instrument your application using zero code for metrics collection, and generate traces for your application. | ## Platforms Customize your platform to work correctly with Kubernetes Monitoring. Expand table | Example | Description | |-------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| | [Azure AKS](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/platforms/azure-aks/README.md) | Enable Kubernetes Monitoring to work correctly with Azure AKS Clusters. | | [EKS Fargate](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/platforms/eks-fargate/README.md) | Gather Pod logs on an EKS Fargate Cluster so that Kubernetes Monitoring works correctly. | | [GKE Autopilot](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/platforms/gke-autopilot/README.md) | Enable Kubernetes Monitoring to work correctly on GKE Autopilot Clusters. | | [OpenShift](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/platforms/openshift/README.md) | Enable Kubernetes Monitoring to work correctly on OpenShift Clusters. | ## Processing and labeling After data collection during the processing phase, you can enable additional processing for telemetry data, refine the metrics you want to keep or drop, and change existing labels or add labels. Expand table | Example | Description | |-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Additional labels](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/extra-rules/README.md) | Use `extraDiscoveryRules` to further refine data collection. | | [Additional processing](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/extra-rules/README.md) | Enable additional processing for logs and metrics, such as `extraMetricProcessingRules` and `extraLogProcessingStages`. | | [Metric enrichment](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/metric-enrichment/README.md) | Enrich metrics with additional labels or metadata. | | [Metrics tuning](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/metrics-tuning/README.md) | Include or exclude metrics to be sent to a metrics destination by using a rule to exclude or include metrics. The default ConfigMap that results from the configuration process creates [allowlists](/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/client-side-filtering/#relabel-prometheus-metrics-to-reduce-usage). These allowlists are configured to keep a subset of metrics used by Kubernetes Monitoring. With metrics tuning, you can add or exclude any metrics from the default allowlist. | | [Tail sampling](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/tail-sampling/README.md) | Configure tail-based sampling policies for traces to control which traces are kept. | To learn more about labels and relabeling, refer to: - [How relabeling in Prometheus works](/blog/2022/03/21/how-relabeling-in-prometheus-works/) - [`Relabel_config` syntax](/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/client-side-filtering/#relabel_config-syntax) To learn more about metrics tuning and allowlists, refer to: - [Reduce usage](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/#reduce-usage) - [Relabel Prometheus metrics to reduce usage](/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/client-side-filtering/#relabel-prometheus-metrics-to-reduce-usage) For examples of labels and annotations during the discovery phase, refer to [Discovery](#discovery). ## Private image registry To support environments that are air-gapped or should be excluded from using public image registries, you can do either of the following: - Globally override the container image registries for every subchart by using a [global object](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/private-image-registries/globally). - Override the individual container image registries for every subchart by using [these values](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/private-image-registries/individual). ## Remote configuration Enable Grafana Alloy to [fetch and load](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/remote-config/README.md) the configuration [from a remote endpoint](/docs/grafana-cloud/send-data/alloy/reference/config-blocks/remotecfg/). ## Scaling and reliability Enhance scaling and reliability where needed. Expand table | Example | Description | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Alloy auto scaling](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/scalability/autoscaling/README.md) | Enable an Alloy instance to scale up and down based on CPU and memory use. | | [Collector storage](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/collector-storage/README.md) | Enable metric scraping to use a Write-Ahead Log (WAL) to store metrics in case of a scrape failure. Enable log gathering to use a volume to store log file positions. This provides a starting point for reading logs after a restart. | | [Highly available kube-state-metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/scalability/high-availability-kube-state-metrics/README.md) | Deploy kube-state-metrics in high availability mode to improve reliability. | | [Resource requests and limits](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/resource-requests-and-limits/README.md) | Set resource requests and limits for the collector workloads. | | [Sharded kube-state-metrics](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/examples/scalability/sharded-kube-state-metrics/README.md) | Shard kube-state-metrics to [improve scaling](https://github.com/kubernetes/kube-state-metrics#scaling-kube-state-metrics). | ## Kubernetes annotations You can target specific namespaces or Pods for data collection with Kubernetes annotations. Often annotations are used for controlling service discovery, but you can also use them to configure how data is collected. ### Annotation autodiscovery Use the [Annotation Autodiscovery feature](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/charts/feature-annotation-autodiscovery) to discover and scrape Prometheus-style metrics from Pods and Services on your Cluster. You can apply these default annotations to a Pod or Service: - `k8s.grafana.com/scrape`: Scrapes Pod or Service for metrics - `k8s.grafana.com/job`: The value to use for the `job` label - `k8s.grafana.com/instance`: The value to use for the `instance` label - `k8s.grafana.com/metrics.container`: The name of the container within the Pod to scrape for metrics. This is used to target a specific container within a Pod that has multiple containers. - `k8s.grafana.com/metrics.path`: The path to scrape for metrics. Defaults to `/metrics`. - `k8s.grafana.com/metrics.portNumber`: The port on the Pod or Service to scrape for metrics. This is used to target a specific port by its number rather than all ports. - `k8s.grafana.com/metrics.portName`: The named port on the Pod or Service to scrape for metrics. This is used to target a specific port by its name rather than all ports. - `k8s.grafana.com/metrics.scheme`: The scheme to use when scraping metrics. Defaults to `http`. - `k8s.grafana.com/metrics.param`: Allows for setting HTTP parameters when calling the scrape endpoint. Use with `k8s.grafana.com/metrics.param_=""`. - `k8s.grafana.com/metrics.scrapeInterval`: The scrape interval to use when scraping metrics. Defaults to `60s`. - `k8s.grafana.com/metrics.scrapeTimeout`: The scrape timeout to use when scraping metrics. Defaults to `10s`. ### Profiling The [Profiling feature](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/charts/feature-profiling) allows you to collect profiling data from your applications. This feature can collect profiles using eBPF, Java, or pprof. #### eBPF profiling To use eBPF to collect CPU profiles from this Pod: `profiles.grafana.com/cpu.ebpf.enabled` #### Java profiling To collect Java profiles from this Pod: `profiles.grafana.com/java.enabled` #### pprof profiling You can use the following annotations to control profiling for each enabled type (`memory`, `block`, `goroutine`, `mutex`, `cpu`, `fgprof`, `godeltaprof_memory`, `godeltaprof_mutex`, and `godeltaprof_block`): - `profiles.grafana.com/.scrape`: Collects pprof profiles for the specified type from this Pod. - `profiles.grafana.com/.port`: Collects profiles for the specified type from this port number. - `profiles.grafana.com/.port_name`: Collects profiles for the specified type from this named port. - `profiles.grafana.com/.path`: Collects profiles for the specified type from this path. - `profiles.grafana.com/.scheme`: The scheme to use when scraping profiles for the specified type. Defaults to `http`. ### Pod logs Use the following annotation to control the collection of [Pod logs](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/charts/feature-pod-logs): `k8s.grafana.com/logs.job`: The value to use for the `job` label ## Extra config You can use the `extraConfig` sections to supply [additional configuration](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/extra-configuration) to the Grafana Alloy instances. Anything you put in these sections are added to the existing configuration that is created by this chart. Helm provides multiple ways to set these additional configuration values. You can either keep the values in the same file as the rest of your Kubernetes Monitoring configuration, or store them separately as their own files and include during Helm chart install. ### Set as values You can set the contents of your extra configuration into your values file: shell ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```shell $ ls values.yaml $ cat values.yaml cluster: name: my-cluster ... collectors: metrics-collector: presets: [clustered, statefulset] extraConfig: |- // Any arbitrary Alloy configuration can be placed here. logging { level = "debug" } logs-collector: presets: [filesystem-log-reader, daemonset] extraConfig: | // Any arbitrary Alloy configuration can be placed here. logging { level = "debug" } ... $ helm upgrade grafana-k8s-monitoring grafana/k8s-monitoring --values values.yaml ``` ### Set as files You can save the contents of your extra configuration as files and use Helm’s `--set-file` argument: shell ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```shell $ ls values.yaml metricsConfig.alloy logsConfig.alloy $ helm upgrade grafana-k8s-monitoring --atomic --timeout 300s grafana/k8s-monitoring \ --values values.yaml \ --set-file "collectors.metrics-collector.extraConfig=metricsConfig.alloy" \ --set-file "collectors.logs-collector.extraConfig=logsConfig.alloy" ``` This can be method beneficial after your extra configuration grows to a certain size.