Customize the Kubernetes Monitoring Helm chart
After you complete the deployment process of the Kubernetes Monitoring Helm chart, you can further customize your configuration. For example, you may want to collect more data, add more destinations, or need guidance on authentication settings.
The examples in the Kubernetes Monitoring Helm chart are complete examples designed to help you correctly alter your configuration by editing the values.yaml file. The following categories provide a means to find the example you need for customization.
After you make any customizations in values.yaml, redeploy the Helm chart to apply your changes.
Helm chart version
Each Kubernetes Monitoring Helm chart version has added functionality. To take advantage of features of an updated version:
- Check the Helm chart release notes for the updates available in each version.
- Install the latest version of the Helm chart by running:
helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring -f values.yaml
If you need to migrate from version 1.x or 2.x, refer to the steps in Version migration.
Authentication
Use the following examples to configure authentication.
Data collection
Alloy
Gather metrics from Grafana Alloy.
Application data
The Application Observability feature encompasses receiving data via various receivers, processing that data, and delivering it to the specified destinations. This example shows the settings to collect telemetry data from an application.
Grafana
Gather metrics and logs from Grafana.
Grafana Loki
Gather metrics and logs from Grafana Loki.
Kubernetes infrastructure data
Monitoring databases that monitor metrics, logs, or traces
With meta monitoring, you can monitor these databases and send the data to Grafana Cloud:
- Enterprise databases: Grafana Enterprise Metrics, Grafana Enterprise Logs, and Grafana Enterprise Traces
- OSS databases: Mimir, Loki, and Tempo
MySQL
Gather metrics and logs from MySQL.
Destinations and proxies
Specify single or multiple destinations, whether a local service deployed on the same cluster, or a remote SaaS service. Use proxy URLs and TLS settings to send data to external services.
Discovery
You can customize data collection during the discovery phase.
For more information on Grafana Alloy labels and relabeling, refer to:
- Alloy
discovery.relabel
to control metrics collection or standardize target labels - Alloy
__meta
labels to refine data collection of Kubernetes resources. An example of this is shown in etcd.
For examples of rules and labeling after the discovery phase, refer to Processing and labeling.
Helm chart deployment
Deploy the Kubernetes Monitoring Helm chart using Terraform.
Instrumentation for applications
Automatically instrument your applications for telemetry collection.
Platforms
Customize your platform to work correctly with Kubernetes Monitoring.
Processing and labeling
After data collection during the processing phase, you can enable additional processing for telemetry data, refine the metrics you want to keep or drop, and change existing labels or add labels.
To learn more about labels and relabeling, refer to:
To learn more about metrics tuning and allowlists, refer to:
For examples of labels and annotations during the discovery phase, refer to Discovery.
Private image registry
To support environments that are air-gapped or should be excluded from using public image registries, you can do either of the following:
- Globally override the container image registries for every subchart by using a global object.
- Override the individual container image registries for every subchart by using these values.
Remote configuration
Enable Grafana Alloy to fetch and load the configuration from a remote endpoint.
Scaling and reliability
Enhance scaling and reliability where needed.
Kubernetes annotations
You can target specific namespaces or Pods for data collection with Kubernetes annotations. Often annotations are used for controlling service discovery, but you can also use them to configure how data is collected.
Annotation autodiscovery
Use the Annotation Autodiscovery feature to discover and scrape Prometheus-style metrics from Pods and Services on your Cluster. You can apply these default annotations to a Pod or Service:
k8s.grafana.com/scrape
: Scrapes Pod or Service for metricsk8s.grafana.com/job
: The value to use for thejob
labelk8s.grafana.com/instance
: The value to use for theinstance
labelk8s.grafana.com/metrics.container
: The name of the container within the Pod to scrape for metrics. This is used to target a specific container within a Pod that has multiple containers.k8s.grafana.com/metrics.path
: The path to scrape for metrics. Defaults to/metrics
.k8s.grafana.com/metrics.portNumber
: The port on the Pod or Service to scrape for metrics. This is used to target a specific port by its number rather than all ports.k8s.grafana.com/metrics.portName
: The named port on the Pod or Service to scrape for metrics. This is used to target a specific port by its name rather than all ports.k8s.grafana.com/metrics.scheme
: The scheme to use when scraping metrics. Defaults tohttp
.k8s.grafana.com/metrics.param
: Allows for setting HTTP parameters when calling the scrape endpoint. Use withk8s.grafana.com/metrics.param_<key>="<value>"
.k8s.grafana.com/metrics.scrapeInterval
: The scrape interval to use when scraping metrics. Defaults to60s
.k8s.grafana.com/metrics.scrapeTimeout
: The scrape timeout to use when scraping metrics. Defaults to10s
.
Profiling
The Profiling feature allows you to collect profiling data from your applications. This feature can collect profiles using eBPF, Java, or pprof.
eBPF profiling
To use eBPF to collect CPU profiles from this Pod:
profiles.grafana.com/cpu.ebpf.enabled
Java profiling
To collect Java profiles from this Pod:
profiles.grafana.com/java.enabled
pprof profiling
You can use the following annotations to control profiling for each enabled type (memory
, block
, goroutine
, mutex
, cpu
, fgprof
, godeltaprof_memory
, godeltaprof_mutex
, and godeltaprof_block
):
profiles.grafana.com/<type>.scrape
: Collects pprof profiles for the specified type from this Pod.profiles.grafana.com/<type>.port
: Collects profiles for the specified type from this port number.profiles.grafana.com/<type>.port_name
: Collects profiles for the specified type from this named port.profiles.grafana.com/<type>.path
: Collects profiles for the specified type from this path.profiles.grafana.com/<type>.scheme
: The scheme to use when scraping profiles for the specified type. Defaults tohttp
.
Pod logs
Use the following annotation to control the collection of Pod logs:
k8s.grafana.com/logs.job
: The value to use for the job
label
Extra config
You can use the extraConfig
sections to supply additional configuration to the Grafana Alloy instances.
Anything you put in these sections are added to the existing configuration that is created by this chart.
Helm provides multiple ways to set these additional configuration values. You can either keep the values in the same file as the rest of your Kubernetes Monitoring configuration, or store them separately as their own files and include during Helm chart install.
Set as values
You can set the contents of your extra configuration into your values file:
$ ls
values.yaml
$ cat values.yaml
cluster:
name: my-cluster
...
alloy-metrics:
extraConfig: |-
// Any arbitrary Alloy configuration can be placed here.
logging {
level = "debug"
}
...
alloy-logs:
...
extraConfig: |
// Any arbitrary Alloy configuration can be placed here.
logging {
level = "debug"
}
...
$ helm upgrade grafana-k8s-monitoring grafana/k8s-monitoring --values values.yaml
Set as files
You can save the contents of your extra configuration as files and use Helm’s --set-file
argument:
$ ls
values.yaml metricsConfig.alloy logsConfig.alloy
$ helm upgrade grafana-k8s-monitoring --atomic --timeout 300s grafana/k8s-monitoring \
--values values.yaml \
--set-file "alloy-metrics.extraConfig=metricsConfig.alloy" \
--set-file "alloy-logs.extraConfig=logsConfig.alloy"
This can be method beneficial after your extra configuration grows to a certain size.