Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Beyla Distributed traces

Distributed traces with Beyla

Introduction

Beyla supports distributed traces for Go applications, both HTTP/S and gRPC, with some limitations and version restrictions.

Go distributed tracing is implemented through the propagation of the W3C traceparent header value. traceparent context propagation is automatic and it doesn’t require any action or configuration.

Beyla will read any incoming trace context header values, track the Go program execution flow and propagate the trace context by automatically adding the traceparent field in outgoing HTTP/gRPC requests. If an application already adds the taceparent field in outgoing requests, Beyla will use that value for tracing instead its own generated trace context. If Beyla cannot find an incoming traceparent context value, it will generate one according to the W3C specification.

Limitations

Kernel integrity mode limitations

In order to write the traceparent value in outgoing HTTP/gRPC request headers, Beyla needs to write to the process memory using the bpf_probe_write_user eBPF helper. Since kernel 5.14 (with fixes backported to the 5.10 series) this helper is protected (and unavailable to BPF programs) if the Linux Kernel is running in integrity lockdown mode. Kernel integrity mode is typically enabled by default if the Kernel has Secure Boot enabled, but it can also be enabled manually.

Beyla will automatically check if it can use the bpf_probe_write_user helper, and enable context propagation only if it’s allowed by the kernel configuration. Verify the Linux Kernel lockdown mode by running the following command:

shell Copy

cat /sys/kernel/security/lockdown

If that file exists and the mode is anything other than [none], Beyla will not be able to perform context propagation and distributed tracing will be disabled.

Configuring distributed tracing for containerized environments (including Kubernetes)

Because of the Kernel lockdown mode restrictions, Docker and Kubernetes configuration files should mount the /sys/kernel/security/ volume for the Beyla docker container from the host system. This way Beyla can correctly determine the Linux Kernel lockdown mode. Here’s an example Docker compose configuration, which ensures Beyla has sufficient information to determine the lockdown mode:

yaml Copy

version: '3.8'

services:
  ...
  beyla:
    image: grafana/beyla:latest
    environment:
      BEYLA_CONFIG_PATH: "/configs/beyla-config.yml"
    volumes:
      - /sys/kernel/security:/sys/kernel/security

If the volume is not mounted, Beyla will assume that the Linux Kernel is not running in integrity mode.

Linux Kernel version limitations for gRPC context propagation (5.17+)

The gRPC protocol headers are typically compressed and encoded with the hpack algorithm. Uncompressed headers are also supported, however gRPC buffers in Go are typically small and writing uncompressed values is not feasible. We’ve implemented hpack encoding in eBPF to support context propagation in Go gRPC, by utilizing the eBPF helper bfp_loop.

The gRPC context propagation for Go requires Linux Kernel versions equal or later than 5.17, because the bpf_loop helper is only available in those kernel versions.