Run Beyla with Cilium
Cilium is an open source project that provides, secures, and observes network connectivity between workloads.
When Grafana Beyla collects network metrics with network.source set to tc, its traffic control eBPF programs can conflict with the ones that Cilium uses.
To prevent the conflict, configure how Beyla and Cilium attach their traffic control programs.
Before you begin
Before you configure Beyla and Cilium, make sure you have the following:
- A Kubernetes cluster that runs Cilium, with Beyla deployed on nodes that meet the Beyla requirements and limitations
kubectl, configured for the cluster- For nodes that run a Linux kernel earlier than 6.6, the
ciliumCLI, with permission to change the Cilium configuration - Access to the Beyla configuration
Configure the attachment mode
Configure Beyla and Cilium so that Beyla programs run before Cilium programs:
Check whether you need to configure the attachment mode. Beyla attaches traffic control programs when you enable network metrics and set
network.sourcetotc. The default fornetwork.sourceissocket_filter, so the default configuration doesn’t attach traffic control programs. If your Beyla configuration doesn’t setnetwork.sourcetotc, you don’t need to configure anything else.Check the Linux kernel version of your nodes:
kubectl get nodes -o wideThe
KERNEL-VERSIONcolumn shows the kernel version of each node. TCX requires Linux kernel 6.6 or later.If your nodes run Linux kernel 6.6 or later, check that the Beyla
ebpf.traffic_control_backendoption or theBEYLA_BPF_TC_BACKENDenvironment variable isautoortcx. The default isauto. With TCX, Beyla programs run before Cilium programs, whether Cilium uses TCX or Netlink. For details, refer to Traffic control backend. If you run Beyla with the Grafana Alloy Helm chart, check thetraffic_control_backendattribute in theebpfblock of thebeyla.ebpfcomponent instead.If your nodes run a Linux kernel earlier than 6.6, set the Cilium Netlink filter priority to 2 or higher:
cilium config set bpf-filter-priority 2Beyla attaches its Netlink filters with priority 1, so they run before Cilium filters with priority 2 or higher. By default,
cilium config setrestarts the Cilium Pods.Check the filter priority in the Cilium configuration:
cilium config view | grep bpf-filter-priorityKeep the Beyla
ebpf.traffic_control_backendoption set toautoortc.Restart Beyla so that it runs the startup check again:
kubectl rollout restart daemonset/<BEYLA_DAEMONSET> --namespace <BEYLA_NAMESPACE>Replace
<BEYLA_DAEMONSET>with the name of the Beyla DaemonSet, and<BEYLA_NAMESPACE>with its namespace. If you run Beyla with the Alloy Helm chart, restart the Alloy DaemonSet instead.Check that the Beyla Pods are running:
kubectl get pods --namespace <BEYLA_NAMESPACE>If you run Beyla with the Alloy Helm chart, the Alloy Pods keep running even when the Cilium check fails, so check the logs in the next step.
Check the logs of a Beyla Pod:
kubectl logs --namespace <BEYLA_NAMESPACE> <POD_NAME>Replace
<POD_NAME>with a Pod name from the previous command. If you run Beyla with the Alloy Helm chart, check the logs of an Alloy Pod. Alloy writes the Beyla logs to its own log, with thesource=beyla-subprocessattribute. If the log doesn’t containCilium compatibility error, Beyla passed the Cilium check. Otherwise, refer to Troubleshoot Cilium compatibility errors.
How Beyla and Cilium share traffic control
Beyla and Cilium use eBPF traffic control classifier programs, BPF_PROG_TYPE_SCHED_CLS.
Beyla attaches these programs when you enable network metrics and set network.source to tc.
These programs attach to the ingress and egress data paths of the kernel networking stack.
Together they form a chain of packet filters.
Each packet filter can inspect the contents of the packet and perform operations, for example redirect or discard the packet.
Beyla programs never disrupt the flow of a packet, but Cilium changes packet flow as part of its operation. If Cilium processes packets before Beyla, it can affect the ability of Beyla to process packets.
Attachment modes
Beyla uses the Traffic Control eXpress (TCX) API or the Netlink interface in the Linux kernel to attach traffic control (TC) programs.
The TCX API is available in Linux kernel 6.6 and later, and is the preferred method to attach TC programs. It provides a linked list mechanism to attach programs to the head, middle, or tail. Beyla and Cilium auto-detect if the kernel supports TCX and use it by default.
When Beyla and Cilium use TCX, they don’t interfere with each other. Beyla attaches its eBPF programs to the head of the list and Cilium to the tail.
The legacy Netlink interface relies on clsact, qdiscs, and a BPF TC filter to attach eBPF programs to network interfaces.
The kernel runs filters in order of their priority.
Filters with lower numbers run first, and 1 is the highest priority.
When TCX isn’t available, both Beyla and Cilium use the Netlink interface to install eBPF programs. If all Cilium TC filters use priority 1, Beyla exits. For the error messages and fixes, refer to Troubleshoot Cilium compatibility errors.
Beyla also exits if you set ebpf.traffic_control_backend to tc and Cilium uses TCX.
Mixed TCX and Netlink
When some programs attach with TCX and others with Netlink, the TCX programs run first. If Beyla uses Netlink and Cilium uses TCX, Cilium programs run before Beyla programs, so Beyla exits.
Troubleshoot Cilium compatibility errors
When the Cilium check fails, Beyla logs wrong Beyla configuration with one of the following errors and exits:
If you run Beyla with Alloy, Alloy logs Beyla subprocess crashed, scheduling restart and restarts Beyla.
If Cilium has priority 1 filters and filters with other priorities, Beyla logs the warning Detected potential Cilium TC left-overs! and keeps running.