Observe and act

Monitor infrastructure

Kubernetes Monitoring

Configure Kubernetes Monitoring

Configure with other methods

OTel with OTel receivers

Grafana Cloud

Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with OTel receivers

To collect Kubernetes telemetry with native OpenTelemetry receivers rather than Prometheus scrape configurations, complete these instructions. The collectors described here accept OTLP from your applications, gather Node and Cluster telemetry through native OTel receivers, collect Pod logs and Cluster events, and send everything to one or more Grafana Cloud stacks.

Note
With this method, Grafana Kubernetes Monitoring supports resource metrics monitoring (CPU and memory), Kubernetes events, and logs. Alerts aren’t supported yet.

To collect Cluster and Node metrics by scraping the kube-state-metrics and node-exporter Prometheus exporters instead, refer to Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with Prometheus exporters.

Before you begin

Before you begin the configuration steps, have the following available:

A Kubernetes Cluster and a kubeconfig with access to it
The kubectl and helm command-line tools installed on your local machine
The OTLP endpoint and basic authentication credentials for each Grafana Cloud stack you send to, which you inject at deploy time rather than storing in the values files. To find these values, refer to Send data to the Grafana Cloud OTLP endpoint.

To add the OpenTelemetry Collector Helm chart repository, run the following commands:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

The two collectors

Cluster-wide receivers must run once, and Node-scoped receivers must run on every Node, so the configuration is split across two Helm releases.

Values file	`mode`	Runs	Logs	Kubernetes events	Other signals
`values-otel-collector.yaml`	`daemonset`	One Pod per Node	Pod and container logs through `filelog`	No, a DaemonSet duplicates them	OTLP metrics, host metrics, and `kubelet` metrics
`values-otel-collector-deployment.yaml`	`deployment`, one replica	One Pod per Cluster	OTLP application logs	Yes, as logs through `k8sobjects`	OTLP metrics and traces, plus `k8s_cluster` metrics

Kubernetes events arrive as logs, not as their own signal type, and only the Deployment collector produces them. For more information, refer to Important components for Kubernetes in the OpenTelemetry documentation.

Warning
Never add the k8s_cluster or k8sobjects receivers to the DaemonSet collector. A DaemonSet duplicates every Cluster metric and event per Node. For more information, refer to Important components for Kubernetes in the OpenTelemetry documentation.

Choose the container image

For the best support from Grafana, we recommend using the grafana/alloy image. We maintain Alloy and can help push changes there, but we don’t maintain the upstream otel/opentelemetry-collector-contrib image.

Both values files use the grafana/alloy image, which runs the collector as the Alloy OTel engine.

image:
  repository: grafana/alloy
  tag: 'v1.18.0'
command:
  name: 'bin/otelcol'

command.name: 'bin/otelcol' is required. Without it, Alloy runs its native engine and ignores the config block.
Pin tag to v1.18.0 or later. The k8s_cluster receiver first ships in v1.18.0-rc.0 and isn’t present in v1.17.1.

To use the upstream collector instead, replace the image block with otel/opentelemetry-collector-contrib and remove the command block.

For more information, refer to Run with the OpenTelemetry Collector Helm chart.

Values files

Create the two files that follow and set your Cluster name where marked REPLACE ME. Endpoints and credentials aren’t stored here, because you inject them at deploy time.

The resources values in these files are a starting point, not a recommendation. Size them to your own workload. Deploy with the values given here, drive representative load, then read actual usage with kubectl -n <NAMESPACE> top pod <COLLECTOR_POD>.

Set each request from the steady-state usage you measure. Kubernetes throttles a container that exceeds its cpu limit, but it terminates a container that exceeds its memory limit. For more information, refer to Resource management for Pods and containers.
Set the memory limit above observed peak, plus roughly 50Mi of headroom. The memory_limiter processor targets the heap only, so it can’t account for the rest of the process.
Keep the CPU limit generous, because exceeding it only throttles the collector rather than terminating it.
Whenever you change limits.memory, keep memory_limiter.limit_mib at approximately 80 percent of it.
Expect the Deployment collector to run heavier than the DaemonSet collector, because k8s_cluster and k8sobjects track every Cluster object and event. Its memory request is higher for that reason.

values-otel-collector.yaml for the DaemonSet collector:

image:
  repository: grafana/alloy
  tag: 'v1.18.0'
command:
  name: 'bin/otelcol'

mode: daemonset

resources:
  limits: { memory: 200Mi, cpu: 200m } #REPLACE THESE NUMBERS
  requests: { memory: 100Mi, cpu: 50m } #REPLACE THESE NUMBERS 

presets:
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true
  hostMetrics:
    enabled: true
  logsCollection:
    enabled: true
    includeCollectorLogs: false

ports:
  prometheus:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    hostmetrics:
      scrapers:
        cpu:
          metrics:
            system.cpu.logical.count:
              enabled: true
        load: {}
        memory:
          metrics:
            system.memory.limit:
              enabled: true
        disk: {}
        filesystem: {}
        network: {}

  processors:
    memory_limiter:
      check_interval: 1s
      limit_mib: 160 # keep at ~80% of resources.limits.memory (REPLACE ME)
      spike_limit_mib: 32 # ~20% of limit_mib (REPLACE ME)
    k8sattributes:
      extract:
        otel_annotations: true
    resource/k8sclustername:
      attributes:
        - key: k8s.cluster.name
          action: upsert
          value: # REPLACE ME with your Cluster name
    resourcedetection:
      detectors: [gcp, system]
      timeout: 2s
      override: true
    resource/hostname:
      attributes:
        - key: k8s.node.name
          action: upsert
          value: '${env:K8S_NODE_NAME}'
        - key: host.name
          action: upsert
          from_attribute: k8s.node.name
    transform/copy_node_name:
      # Workaround: promote k8s.node.name and host.name to datapoint attributes so the Grafana Cloud Prometheus OTLP endpoint keeps them as labels
      metric_statements:
        - context: datapoint
          statements:
            - set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"]) where resource.attributes["k8s.node.name"] != nil
            - set(attributes["host.name"], resource.attributes["host.name"]) where resource.attributes["host.name"] != nil
    batch: {}

  exporters:
    otlphttp/grafanaCloudOTLPEndpoint:
      auth:
        authenticator: basicauth/grafanaCloudOTLPEndpoint
    prometheus:
      endpoint: '0.0.0.0:8889'
      resource_to_telemetry_conversion:
        enabled: true

  service:
    extensions:
      - basicauth/grafanaCloudOTLPEndpoint
      - health_check
    pipelines:
      logs:
        processors: [memory_limiter, k8sattributes, resource/k8sclustername, batch]
        exporters: [otlphttp/grafanaCloudOTLPEndpoint]
      metrics:
        receivers: [otlp, hostmetrics]
        # Order matters: processors run in the order listed, so detect, then set attributes, then copy, then batch
        processors: [memory_limiter, resourcedetection, k8sattributes, resource/k8sclustername, resource/hostname, transform/copy_node_name, batch]
        exporters: [otlphttp/grafanaCloudOTLPEndpoint, prometheus]

The logs pipeline lists no receivers because the logsCollection preset adds a filelog receiver for Pod and container logs automatically. To collect OTLP logs on this collector as well, add otlp to that pipeline’s receivers.

values-otel-collector-deployment.yaml for the Deployment collector:

image:
  repository: grafana/alloy
  tag: 'v1.18.0'
command:
  name: 'bin/otelcol'

mode: deployment
replicaCount: 1
# A single-Node Cluster can't fit a surge Pod during a RollingUpdate, so recreate instead
rollout:
  strategy: Recreate

resources:
  limits: { memory: 200Mi, cpu: 200m } #REPLACE THESE NUMBERS
  requests: { memory: 150Mi, cpu: 50m } #REPLACE THESE NUMBERS

presets:
  kubernetesAttributes:
    enabled: true
  clusterMetrics:
    enabled: true

ports:
  prometheus:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    k8s_cluster:
      metrics:
        k8s.container.status.reason:
          enabled: true
    k8sobjects:
      objects:
        - name: events
          mode: watch

  processors:
    memory_limiter:
      check_interval: 1s
      limit_mib: 160 # keep at ~80% of resources.limits.memory (REPLACE ME)
      spike_limit_mib: 32 # ~20% of limit_mib (REPLACE ME)
    k8sattributes:
      extract:
        otel_annotations: true
    resource/k8sclustername:
      attributes:
        - key: k8s.cluster.name
          action: upsert
          value: # REPLACE ME with your Cluster name
    resource/hostname:
      attributes:
        - key: host.name
          action: upsert
          from_attribute: k8s.node.name
    resource/eventservicename:
      attributes:
        - key: service.name
          action: upsert
          value: 'integrations/kubernetes/eventhandler'
    transform/copy_node_name:
      # Workaround: promote k8s.node.name and host.name to datapoint attributes so the Grafana Cloud Prometheus OTLP endpoint keeps them as labels
      metric_statements:
        - context: datapoint
          statements:
            - set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"]) where resource.attributes["k8s.node.name"] != nil
            - set(attributes["host.name"], resource.attributes["host.name"]) where resource.attributes["host.name"] != nil
    batch: {}

  exporters:
    otlphttp/grafanaCloudOTLPEndpoint:
      auth:
        authenticator: basicauth/grafanaCloudOTLPEndpoint
    prometheus:
      endpoint: '0.0.0.0:8889'
      resource_to_telemetry_conversion:
        enabled: true

  service:
    extensions:
      - basicauth/grafanaCloudOTLPEndpoint
      - health_check
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, resource/k8sclustername, batch]
        exporters: [otlphttp/grafanaCloudOTLPEndpoint]
      logs/events:
        receivers: [k8sobjects]
        processors: [memory_limiter, k8sattributes, resource/k8sclustername, resource/eventservicename, batch]
        exporters: [otlphttp/grafanaCloudOTLPEndpoint]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, resource/k8sclustername, batch]
        exporters: [otlphttp/grafanaCloudOTLPEndpoint]
      metrics:
        receivers: [otlp, k8s_cluster]
        # Order matters: processors run in the order listed, so set attributes, then copy, then batch
        processors: [memory_limiter, k8sattributes, resource/k8sclustername, resource/hostname, transform/copy_node_name, batch]
        exporters: [otlphttp/grafanaCloudOTLPEndpoint, prometheus]

Events flow on their own logs/events pipeline so that resource/eventservicename stamps service.name=integrations/kubernetes/eventhandler, which is the value the event views in Kubernetes Monitoring match, without touching your OTLP application logs.

Key settings

Both files repeat most of their configuration. The following settings are the ones to understand before you change anything.

presets.kubernetesAttributes configures the k8sattributes processor, which enriches telemetry with Pod and namespace metadata. For more information, refer to Configuration for Kubernetes attributes processor.
The otlp receiver accepts OTLP over gRPC on port 4317 and HTTP on port 4318. The 0.0.0.0 endpoint lets other Pods reach it. For more information, refer to the OTLP receiver documentation.
The memory_limiter processor protects the collector from running out of memory. Keep limit_mib below the Pod memory limit.
The k8sattributes processor extracts Kubernetes metadata and OpenTelemetry Pod annotations.
The resource/k8sclustername processor stamps k8s.cluster.name. Replace its REPLACE ME value with your Cluster name.
In the hostmetrics receiver, each block is a host subsystem and {} means the scraper’s default metrics. The two metrics enabled explicitly are off by default, and they report the Node’s total memory and logical CPU count, which you need to read raw usage as a utilization percentage. For the full metric list, refer to the host metrics receiver documentation.
Extensions stay inactive until you list them under service.extensions. An exporter that references an unlisted authenticator fails to start.
ports.prometheus exposes port 8889 and must match the prometheus exporter endpoint.

Deploy the collectors

Install each collector as a separate Helm release, passing the endpoint and credentials at deploy time. To send to more than one stack, define one otlphttp/<NAME> exporter and a matching basicauth/<NAME> extension per destination, and repeat the three --set flags for each one.

helm upgrade --install alloy-otel-collector open-telemetry/opentelemetry-collector \
  -n default -f values-otel-collector.yaml \
  --set config.exporters.otlphttp/grafanaCloudOTLPEndpoint.endpoint=<OTLP_URL> \
  --set-string config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.username=<USERNAME> \
  --set config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.password=<PASSWORD>

helm upgrade --install alloy-otel-collector-deployment open-telemetry/opentelemetry-collector \
  -n default -f values-otel-collector-deployment.yaml \
  --set config.exporters.otlphttp/grafanaCloudOTLPEndpoint.endpoint=<OTLP_URL> \
  --set-string config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.username=<USERNAME> \
  --set config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.password=<PASSWORD>

Use --set-string for usernames so values that look numeric aren’t converted to numbers.

Verify

To confirm both collectors are running, run the following command:

kubectl -n default get pods

To check that logs and events are arriving, open Explore in Grafana Cloud and select your logs data source. In the following queries, replace <YOUR_CLUSTER_NAME> with the name you set in resource/k8sclustername.

To query your Pod and application logs, run the following LogQL query:

{k8s_cluster_name="<YOUR_CLUSTER_NAME>", service_name!="integrations/kubernetes/eventhandler"}

To query the Kubernetes events, run the following LogQL query:

{k8s_cluster_name="<YOUR_CLUSTER_NAME>", service_name="integrations/kubernetes/eventhandler"}

The k8sobjects receiver runs in watch mode, so only events that occur after the Deployment collector Pod starts appear. If the events stream looks empty, cause some Cluster activity, such as restarting a Pod, then run the query again.

After these steps, you can see your resources and metrics in Kubernetes Monitoring.

Was this page helpful?

Email docs@grafana.com

Help and support

Community

Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with OTel receivers

Before you begin

The two collectors

Choose the container image

Values files

Key settings

Deploy the collectors

Verify

Was this page helpful?

Still have questions?

Get every update

Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with OTel receivers

Before you begin

The two collectors

Choose the container image

Values files

Key settings

Deploy the collectors

Verify

Was this page helpful?

Related resources from Grafana Labs