--- title: "How SDK injection works | Grafana Cloud documentation" description: "How Instrumentation Hub uses SDK injection to instrument Kubernetes application services without code changes" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # How SDK injection works SDK injection adds OpenTelemetry SDK instrumentation to your Kubernetes workloads when their Pods start. You don’t need to change application code, rebuild container images, or edit workload manifests. Like the rest of Instrumentation Hub, SDK injection currently supports Kubernetes Linux environments only. When you instrument application services with [Instrumentation Hub](../), Grafana Cloud uses two techniques together: - **eBPF instrumentation** from [Beyla](/docs/beyla/latest/) observes HTTP, gRPC, and database calls from outside the process, in any language. - **SDK injection** loads the OpenTelemetry SDK into the process itself, which adds spans from the libraries and frameworks your application uses. The two techniques coordinate so you don’t get duplicate telemetry. When Beyla can generate span metrics for a Pod, the injected SDK skips generating them. You don’t declare a language per workload, instead Grafana Cloud ships one image that carries the instrumentation for every supported language, and the injector picks the right one at runtime. SDK injection supports .NET, Java, Node.js, and Python. ## How configuration reaches your Pods Your choices in Instrumentation Hub reach your Pods through Fleet Management, Alloy, and the SDK injector. Nothing in this path requires you to run a command or edit a manifest. 1. You select services and pipeline options in Instrumentation Hub and start instrumentation. 2. Instrumentation Hub publishes a configuration pipeline to [Fleet Management](/docs/grafana-cloud/send-data/fleet-management/). 3. Alloy in your Cluster fetches the pipeline. The pipeline sets the OTLP endpoint the injected SDK exports to, the trace propagators, the sampling rate, and which signals the SDK sends. 4. Alloy writes that configuration into ConfigMaps in its own namespace. Each ConfigMap holds an ordered list of rules that match Pods by namespace, owner, labels, and annotations, and a list of workloads that are eligible for restart. 5. The SDK injector watches those ConfigMaps and applies the matching rule the next time a Pod is created. Rules are evaluated in order and the first match wins, so a rule can also exclude a Pod from instrumentation. This is how Instrumentation Hub instruments a whole namespace while leaving individual services out. A validating webhook allows only Alloy to write these ConfigMaps, so nothing else in your Cluster can change what gets instrumented. ## What the injector changes in your Pods The injector runs as a single Deployment for the whole Cluster, not one per Node. It’s installed with Alloy, so installing it needs permission to create cluster-scoped resources. Refer to [Install Grafana Alloy](../#install-grafana-alloy). A mutating admission webhook intercepts Pod creation and, when a rule matches, makes three changes to every container in the Pod: - Mounts a read-only volume that holds the OpenTelemetry SDK payload. - Sets `LD_PRELOAD` to a small library in that volume. As your application starts, that library sets the variable the runtime uses to load an agent, such as `JAVA_TOOL_OPTIONS` for a JVM or `DOTNET_STARTUP_HOOKS` for .NET. It sets these variables for every supported language, and only the one your runtime reads has any effect. - Adds the OTLP and resource attribute environment variables the SDK needs, including the service name, service namespace, service version, and the Kubernetes Pod, container, and Node identifiers. The injector delivers the SDK payload in one of two ways, and chooses between them at startup based on your Cluster’s Kubernetes version: Expand table | Mode | Kubernetes version | How the payload arrives | |----------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------| | Image volume | 1.35 and later | The Pod mounts the SDK image directly as an image volume. Nothing is copied. | | Init container | Earlier than 1.35 | The Pod gets an ephemeral volume with a 500 MiB limit, and an init container copies the SDK payload into it before your containers start. | Injection is idempotent. The injector records which configuration a Pod was instrumented with, so a Pod that already runs the current configuration is left alone. ## Why Pods restart `LD_PRELOAD` takes effect only when a process starts, so a Pod that’s already running has to restart before it’s instrumented. The injector handles this for you. When Alloy publishes a new configuration, the injector evicts the Pods that both match a rule and appear in the eligible for restart list that Beyla built while discovering your Cluster. Because Beyla only lists workloads it identified as eligible for instrumentation, services in unsupported languages aren’t restarted. Evictions honor `PodDisruptionBudgets`, and the injector skips bare Pods that no workload controller owns, so it never evicts a Pod that nothing would recreate. For replicated workloads, the restart is a rolling replacement rather than downtime. Pods also restart when the instrumentation configuration changes: - Changing a pipeline setting, such as the trace sampling rate or which signals you collect, produces a new configuration. Eligible Pods restart to pick it up. - Grafana Cloud upgrades the injected SDK when a new version is available, for example to address a CVE or add a feature. The injector tracks the SDK version in each Pod, so an upgrade produces a new configuration and eligible Pods restart to pick it up. - Removing instrumentation restarts Pods so the injected volume and environment variables are stripped back out. Because the injector reacts to configuration rather than to a one time action, you can reverse a pipeline change by switching the setting back. Refer to [Manage instrumentation and settings](../#manage-instrumentation-and-settings). ## Applications the injector skips Loading an SDK into a running application can break it, so the injector and the injected payload both check for conflicts and step aside when they find one. An application that can’t take SDK injection keeps running and falls back to eBPF instrumentation. The injector skips a Pod entirely, and doesn’t evict it, when: - A container already sets `LD_PRELOAD` to something else. Overwriting it would break the application. - The Pod already runs the current instrumentation configuration. - The Pod is in a namespace the webhook never touches, such as `kube-system`, `kube-public`, `kube-node-lease`, or the namespaces that hold Alloy and the injector itself. After injection, the SDK payload runs its own checks inside the container and doesn’t initialize when: - The runtime version is too old to support the SDK, for example Python earlier than 3.9. - An incompatible version of an OpenTelemetry package is already installed, which is a common source of breakage in Python applications. - The application already loads an OpenTelemetry SDK itself, either as an installed dependency or through a startup argument. Your own instrumentation takes precedence. - The application is a native binary, such as a GraalVM native image or a .NET Native AOT build. There’s no runtime to load an agent into, so the injected library does nothing. The admission webhook is also configured to fail open. If the injector is unavailable, your Pods are created without SDK injection rather than failing to start, and they still receive eBPF instrumentation. ## How to tell whether a Pod is instrumented The injector annotates every Pod it instruments with `beyla.grafana.com/inject`. The value identifies the configuration the Pod was instrumented with, so it changes when a pipeline setting or the SDK version changes. To check a Pod, run: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl get pod --namespace -o jsonpath='{.metadata.annotations.beyla\.grafana\.com/inject}' ``` An empty result means the Pod isn’t instrumented. If that’s a Pod you expected to be instrumented, refer to [Applications the injector skips](#applications-the-injector-skips). ## Next steps - [Instrument applications and services](../#instrument-applications-and-services) in Instrumentation Hub. - Review your coverage with [instrumentation quality](/docs/grafana-cloud/monitor-applications/application-observability/setup/instrumentation-quality/) in Application Observability. - Tune injection settings directly in Alloy with the [`injector` block](/docs/grafana-cloud/observe-and-act/send-data/alloy/reference/components/beyla/beyla.ebpf/#injector) of the `beyla.ebpf` component.