Distributed traces with Beyla
Beyla supports distributed traces for Go applications, both HTTP/S and gRPC, with some limitations and version restrictions.
Go distributed tracing is implemented through the propagation of the W3C
traceparent header value.
traceparent context propagation is automatic and it doesn’t require any action or configuration.
Beyla will read any incoming trace context header values, track the Go program execution flow and propagate the trace context by automatically adding the
traceparent field in outgoing HTTP/gRPC requests. If an application already adds the
taceparent field in outgoing requests, Beyla will use that value for tracing instead its own generated trace context. If Beyla cannot find an incoming
traceparent context value, it will generate one according to the W3C specification.
Kernel integrity mode limitations
In order to write the
traceparent value in outgoing HTTP/gRPC request headers, Beyla needs to write to the process memory using the bpf_probe_write_user eBPF helper. Since kernel 5.14 (with fixes backported to the 5.10 series) this helper is protected (and unavailable to BPF programs) if the Linux Kernel is running in
integrity lockdown mode. Kernel integrity mode is typically enabled by default if the Kernel has Secure Boot enabled, but it can also be enabled manually.
Beyla will automatically check if it can use the
bpf_probe_write_user helper, and enable context propagation only if it’s allowed by the kernel configuration. Verify the Linux Kernel lockdown mode by running the following command:
If that file exists and the mode is anything other than
[none], Beyla will not be able to perform context propagation and distributed tracing will be disabled.
Configuring distributed tracing for containerized environments (including Kubernetes)
Because of the Kernel lockdown mode restrictions, Docker and Kubernetes configuration files should mount the
/sys/kernel/security/ volume for the Beyla docker container from the host system. This way Beyla can correctly determine the Linux Kernel lockdown mode. Here’s an example Docker compose configuration, which ensures Beyla has sufficient information to determine the lockdown mode:
If the volume is not mounted, Beyla will assume that the Linux Kernel is not running in integrity mode.
Linux Kernel version limitations for gRPC context propagation (5.17+)
The gRPC protocol headers are typically compressed and encoded with the
hpack algorithm. Uncompressed headers are also supported, however gRPC buffers in Go are typically small and writing uncompressed values is not feasible. We’ve implemented
hpack encoding in eBPF to support context propagation in Go gRPC, by utilizing the eBPF helper bfp_loop.
The gRPC context propagation for Go requires Linux Kernel versions equal or later than 5.17, because the
bpf_loop helper is only available in those kernel versions.