--- title: "Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with OTel receivers | Grafana Cloud documentation" description: "Configure the OpenTelemetry Collector with OTel receivers to send Kubernetes metrics, logs, and events to Grafana Cloud to view in Grafana Kubernetes Monitoring" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with OTel receivers To collect Kubernetes telemetry with native OpenTelemetry receivers rather than Prometheus scrape configurations, complete these instructions. The collectors described here accept OTLP from your applications, gather Node and Cluster telemetry through native OTel receivers, collect Pod logs and Cluster events, and send everything to one or more Grafana Cloud stacks. > Note > > With this method, Grafana Kubernetes Monitoring supports resource metrics monitoring (CPU and memory), Kubernetes events, and logs. Alerts aren’t supported yet. To collect Cluster and Node metrics by scraping the `kube-state-metrics` and `node-exporter` Prometheus exporters instead, refer to [Send Kubernetes metrics, logs, and events using the OpenTelemetry Collector with Prometheus exporters](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/config-other-methods/otel-collector/). ## Before you begin Before you begin the configuration steps, have the following available: - A Kubernetes Cluster and a `kubeconfig` with access to it - The [`kubectl`](https://kubernetes.io/docs/tasks/tools/) and `helm` command-line tools installed on your local machine - The OTLP endpoint and basic authentication credentials for each Grafana Cloud stack you send to, which you inject at deploy time rather than storing in the values files. To find these values, refer to [Send data to the Grafana Cloud OTLP endpoint](/docs/grafana-cloud/observe-and-act/send-data/otlp/send-data-otlp/). To add the OpenTelemetry Collector Helm chart repository, run the following commands: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update ``` ## The two collectors Cluster-wide receivers must run once, and Node-scoped receivers must run on every Node, so the configuration is split across two Helm releases. Expand table | Values file | `mode` | Runs | Logs | Kubernetes events | Other signals | |-----------------------------------------|---------------------------|---------------------|------------------------------------------|-----------------------------------|-----------------------------------------------------| | `values-otel-collector.yaml` | `daemonset` | One Pod per Node | Pod and container logs through `filelog` | No, a DaemonSet duplicates them | OTLP metrics, host metrics, and `kubelet` metrics | | `values-otel-collector-deployment.yaml` | `deployment`, one replica | One Pod per Cluster | OTLP application logs | Yes, as logs through `k8sobjects` | OTLP metrics and traces, plus `k8s_cluster` metrics | Kubernetes events arrive as logs, not as their own signal type, and only the Deployment collector produces them. For more information, refer to [Important components for Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/collector/components/) in the OpenTelemetry documentation. > Warning > > Never add the `k8s_cluster` or `k8sobjects` receivers to the DaemonSet collector. A DaemonSet duplicates every Cluster metric and event per Node. For more information, refer to [Important components for Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/collector/components/) in the OpenTelemetry documentation. ## Choose the container image For the best support from Grafana, we recommend using the `grafana/alloy` image. We maintain Alloy and can help push changes there, but we don’t maintain the upstream `otel/opentelemetry-collector-contrib` image. Both values files use the `grafana/alloy` image, which runs the collector as the Alloy OTel engine. YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml image: repository: grafana/alloy tag: 'v1.18.0' command: name: 'bin/otelcol' ``` - `command.name: 'bin/otelcol'` is required. Without it, Alloy runs its native engine and ignores the `config` block. - Pin `tag` to `v1.18.0` or later. The `k8s_cluster` receiver first ships in v1.18.0-rc.0 and isn’t present in v1.17.1. To use the upstream collector instead, replace the `image` block with `otel/opentelemetry-collector-contrib` and remove the `command` block. For more information, refer to [Run with the OpenTelemetry Collector Helm chart](/docs/grafana-cloud/observe-and-act/send-data/alloy/set-up/otel_engine/#run-with-the-opentelemetry-collector-helm-chart). ## Values files Create the two files that follow and set your Cluster name where marked `REPLACE ME`. Endpoints and credentials aren’t stored here, because you inject them at deploy time. The `resources` values in these files are a starting point, not a recommendation. Size them to your own workload. Deploy with the values given here, drive representative load, then read actual usage with `kubectl -n top pod `. - Set each request from the steady-state usage you measure. Kubernetes throttles a container that exceeds its `cpu` limit, but it terminates a container that exceeds its `memory` limit. For more information, refer to [Resource management for Pods and containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). - Set the memory limit above observed peak, plus roughly 50Mi of headroom. The [`memory_limiter` processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md) targets the heap only, so it can’t account for the rest of the process. - Keep the CPU limit generous, because exceeding it only throttles the collector rather than terminating it. - Whenever you change `limits.memory`, keep `memory_limiter.limit_mib` at approximately 80 percent of it. - Expect the Deployment collector to run heavier than the DaemonSet collector, because `k8s_cluster` and `k8sobjects` track every Cluster object and event. Its memory request is higher for that reason. `values-otel-collector.yaml` for the DaemonSet collector: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml image: repository: grafana/alloy tag: 'v1.18.0' command: name: 'bin/otelcol' mode: daemonset resources: limits: { memory: 200Mi, cpu: 200m } #REPLACE THESE NUMBERS requests: { memory: 100Mi, cpu: 50m } #REPLACE THESE NUMBERS presets: kubernetesAttributes: enabled: true kubeletMetrics: enabled: true hostMetrics: enabled: true logsCollection: enabled: true includeCollectorLogs: false ports: prometheus: enabled: true containerPort: 8889 servicePort: 8889 protocol: TCP config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 hostmetrics: scrapers: cpu: metrics: system.cpu.logical.count: enabled: true load: {} memory: metrics: system.memory.limit: enabled: true disk: {} filesystem: {} network: {} processors: memory_limiter: check_interval: 1s limit_mib: 160 # keep at ~80% of resources.limits.memory (REPLACE ME) spike_limit_mib: 32 # ~20% of limit_mib (REPLACE ME) k8sattributes: extract: otel_annotations: true resource/k8sclustername: attributes: - key: k8s.cluster.name action: upsert value: # REPLACE ME with your Cluster name resourcedetection: detectors: [gcp, system] timeout: 2s override: true resource/hostname: attributes: - key: k8s.node.name action: upsert value: '${env:K8S_NODE_NAME}' - key: host.name action: upsert from_attribute: k8s.node.name transform/copy_node_name: # Workaround: promote k8s.node.name and host.name to datapoint attributes so the Grafana Cloud Prometheus OTLP endpoint keeps them as labels metric_statements: - context: datapoint statements: - set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"]) where resource.attributes["k8s.node.name"] != nil - set(attributes["host.name"], resource.attributes["host.name"]) where resource.attributes["host.name"] != nil batch: {} exporters: otlphttp/grafanaCloudOTLPEndpoint: auth: authenticator: basicauth/grafanaCloudOTLPEndpoint prometheus: endpoint: '0.0.0.0:8889' resource_to_telemetry_conversion: enabled: true service: extensions: - basicauth/grafanaCloudOTLPEndpoint - health_check pipelines: logs: processors: [memory_limiter, k8sattributes, resource/k8sclustername, batch] exporters: [otlphttp/grafanaCloudOTLPEndpoint] metrics: receivers: [otlp, hostmetrics] # Order matters: processors run in the order listed, so detect, then set attributes, then copy, then batch processors: [memory_limiter, resourcedetection, k8sattributes, resource/k8sclustername, resource/hostname, transform/copy_node_name, batch] exporters: [otlphttp/grafanaCloudOTLPEndpoint, prometheus] ``` The `logs` pipeline lists no receivers because the `logsCollection` preset adds a `filelog` receiver for Pod and container logs automatically. To collect OTLP logs on this collector as well, add `otlp` to that pipeline’s receivers. `values-otel-collector-deployment.yaml` for the Deployment collector: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml image: repository: grafana/alloy tag: 'v1.18.0' command: name: 'bin/otelcol' mode: deployment replicaCount: 1 # A single-Node Cluster can't fit a surge Pod during a RollingUpdate, so recreate instead rollout: strategy: Recreate resources: limits: { memory: 200Mi, cpu: 200m } #REPLACE THESE NUMBERS requests: { memory: 150Mi, cpu: 50m } #REPLACE THESE NUMBERS presets: kubernetesAttributes: enabled: true clusterMetrics: enabled: true ports: prometheus: enabled: true containerPort: 8889 servicePort: 8889 protocol: TCP config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 k8s_cluster: metrics: k8s.container.status.reason: enabled: true k8sobjects: objects: - name: events mode: watch processors: memory_limiter: check_interval: 1s limit_mib: 160 # keep at ~80% of resources.limits.memory (REPLACE ME) spike_limit_mib: 32 # ~20% of limit_mib (REPLACE ME) k8sattributes: extract: otel_annotations: true resource/k8sclustername: attributes: - key: k8s.cluster.name action: upsert value: # REPLACE ME with your Cluster name resource/hostname: attributes: - key: host.name action: upsert from_attribute: k8s.node.name resource/eventservicename: attributes: - key: service.name action: upsert value: 'integrations/kubernetes/eventhandler' transform/copy_node_name: # Workaround: promote k8s.node.name and host.name to datapoint attributes so the Grafana Cloud Prometheus OTLP endpoint keeps them as labels metric_statements: - context: datapoint statements: - set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"]) where resource.attributes["k8s.node.name"] != nil - set(attributes["host.name"], resource.attributes["host.name"]) where resource.attributes["host.name"] != nil batch: {} exporters: otlphttp/grafanaCloudOTLPEndpoint: auth: authenticator: basicauth/grafanaCloudOTLPEndpoint prometheus: endpoint: '0.0.0.0:8889' resource_to_telemetry_conversion: enabled: true service: extensions: - basicauth/grafanaCloudOTLPEndpoint - health_check pipelines: logs: receivers: [otlp] processors: [memory_limiter, k8sattributes, resource/k8sclustername, batch] exporters: [otlphttp/grafanaCloudOTLPEndpoint] logs/events: receivers: [k8sobjects] processors: [memory_limiter, k8sattributes, resource/k8sclustername, resource/eventservicename, batch] exporters: [otlphttp/grafanaCloudOTLPEndpoint] traces: receivers: [otlp] processors: [memory_limiter, k8sattributes, resource/k8sclustername, batch] exporters: [otlphttp/grafanaCloudOTLPEndpoint] metrics: receivers: [otlp, k8s_cluster] # Order matters: processors run in the order listed, so set attributes, then copy, then batch processors: [memory_limiter, k8sattributes, resource/k8sclustername, resource/hostname, transform/copy_node_name, batch] exporters: [otlphttp/grafanaCloudOTLPEndpoint, prometheus] ``` Events flow on their own `logs/events` pipeline so that `resource/eventservicename` stamps `service.name=integrations/kubernetes/eventhandler`, which is the value the event views in Kubernetes Monitoring match, without touching your OTLP application logs. ## Key settings Both files repeat most of their configuration. The following settings are the ones to understand before you change anything. - `presets.kubernetesAttributes` configures the `k8sattributes` processor, which enriches telemetry with Pod and namespace metadata. For more information, refer to [Configuration for Kubernetes attributes processor](https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/README.md#configuration-for-kubernetes-attributes-processor). - The `otlp` receiver accepts OTLP over gRPC on port `4317` and HTTP on port `4318`. The `0.0.0.0` endpoint lets other Pods reach it. For more information, refer to the [OTLP receiver documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver). - The `memory_limiter` processor protects the collector from running out of memory. Keep `limit_mib` below the Pod memory limit. - The `k8sattributes` processor extracts Kubernetes metadata and OpenTelemetry Pod annotations. - The `resource/k8sclustername` processor stamps `k8s.cluster.name`. Replace its `REPLACE ME` value with your Cluster name. - In the `hostmetrics` receiver, each block is a host subsystem and `{}` means the scraper’s default metrics. The two metrics enabled explicitly are off by default, and they report the Node’s total memory and logical CPU count, which you need to read raw usage as a utilization percentage. For the full metric list, refer to the [host metrics receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/README.md). - Extensions stay inactive until you list them under `service.extensions`. An exporter that references an unlisted authenticator fails to start. - `ports.prometheus` exposes port `8889` and must match the `prometheus` exporter endpoint. ## Deploy the collectors Install each collector as a separate Helm release, passing the endpoint and credentials at deploy time. To send to more than one stack, define one `otlphttp/` exporter and a matching `basicauth/` extension per destination, and repeat the three `--set` flags for each one. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm upgrade --install alloy-otel-collector open-telemetry/opentelemetry-collector \ -n default -f values-otel-collector.yaml \ --set config.exporters.otlphttp/grafanaCloudOTLPEndpoint.endpoint= \ --set-string config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.username= \ --set config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.password= helm upgrade --install alloy-otel-collector-deployment open-telemetry/opentelemetry-collector \ -n default -f values-otel-collector-deployment.yaml \ --set config.exporters.otlphttp/grafanaCloudOTLPEndpoint.endpoint= \ --set-string config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.username= \ --set config.extensions.basicauth/grafanaCloudOTLPEndpoint.client_auth.password= ``` Use `--set-string` for usernames so values that look numeric aren’t converted to numbers. ## Verify To confirm both collectors are running, run the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl -n default get pods ``` To check that logs and events are arriving, open **Explore** in Grafana Cloud and select your logs data source. In the following queries, replace `` with the name you set in `resource/k8sclustername`. To query your Pod and application logs, run the following LogQL query: logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```logql {k8s_cluster_name="", service_name!="integrations/kubernetes/eventhandler"} ``` To query the Kubernetes events, run the following LogQL query: logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```logql {k8s_cluster_name="", service_name="integrations/kubernetes/eventhandler"} ``` The `k8sobjects` receiver runs in `watch` mode, so only events that occur after the Deployment collector Pod starts appear. If the events stream looks empty, cause some Cluster activity, such as restarting a Pod, then run the query again. After these steps, you can see your resources and metrics in Kubernetes Monitoring.