Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentationbreadcrumb arrow Beylabreadcrumb arrow Setupbreadcrumb arrow Deploy in Kubernetes
Grafana Cloud

Deploy Beyla in Kubernetes

Note

This document explains how to manually deploy Beyla in Kubernetes, setting up all the required entities by yourself.

You might prefer to follow the Deploy Beyla in Kubernetes with Helm documentation instead.

Contents:

Configuring Kubernetes metadata decoration

Beyla can decorate your traces with the following Kubernetes labels:

  • k8s.namespace.name
  • k8s.deployment.name
  • k8s.statefulset.name
  • k8s.replicaset.name
  • k8s.daemonset.name
  • k8s.node.name
  • k8s.pod.name
  • k8s.pod.uid
  • k8s.pod.start_time

To enable metadata decoration, you need to:

  • Create a ServiceAccount and bind a ClusterRole granting list and watch permissions for both Pods and ReplicaSets. You can do it by deploying this example file:
yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: beyla
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: beyla
rules:
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: beyla
subjects:
  - kind: ServiceAccount
    name: beyla
    namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: beyla

(You need to change the namespace: default value if you are deploying Beyla in another namespace).

  1. Configure Beyla with the BEYLA_KUBE_METADATA_ENABLE=true environment variable, or the attributes.kubernetes.enable: true YAML configuration.

  2. Don’t forget to specify the serviceAccountName: beyla property in your Beyla Pod (as shown in the later deployment examples).

Optionally, select which Kubernetes services to instrument in the discovery -> services section of the YAML configuration file. For more information, refer to the Service discovery section in the Configuration document, as well as the Providing an external configuration file section of this page.

Deploying Beyla

You can deploy Beyla in Kubernetes in two different ways:

  • As a sidecar container
  • As a DaemonSet

Deploy Beyla as a sidecar container

This is the way you can deploy Beyla if you want to monitor a given service that might not be deployed in all the hosts, so you only have to deploy one Beyla instance per each service instance.

Deploying Beyla as a sidecar container has the following configuration requirements:

  • The process namespace must be shared between all containers in the Pod (shareProcessNamespace: true pod variable)
  • The auto-instrument container must run in privileged mode (securityContext.privileged: true property of the container configuration).
    • Some Kubernetes installation allow the following securityContext configuration, but it might not work with all the container runtime configurations, as some of them confine the containers and remove some permissions:
      yaml 
      securityContext:
  runAsUser: 0
  capabilities:
    add:
      - SYS_ADMIN
      - SYS_RESOURCE # not required for kernels 5.11+

The following example instruments the goblog pod by attaching Beyla as a container (image available at grafana/beyla:latest). The auto-instrumentation tool is configured to forward metrics and traces to Grafana Alloy, which is accessible behind the grafana-alloy service in the same namespace:

yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: goblog
  labels:
    app: goblog
spec:
  replicas: 2
  selector:
    matchLabels:
      app: goblog
  template:
    metadata:
      labels:
        app: goblog
    spec:
      # Required so the sidecar instrument tool can access the service process
      shareProcessNamespace: true
      serviceAccountName: beyla # required if you want kubernetes metadata decoration
      containers:
        # Container for the instrumented service
        - name: goblog
          image: mariomac/goblog:dev
          imagePullPolicy: IfNotPresent
          command: ["/goblog"]
          env:
            - name: "GOBLOG_CONFIG"
              value: "/sample/config.yml"
          ports:
            - containerPort: 8443
              name: https
        # Sidecar container with Beyla - the eBPF auto-instrumentation tool
        - name: beyla
          image: grafana/beyla:latest
          securityContext: # Privileges are required to install the eBPF probes
            privileged: true
          env:
            # The internal port of the goblog application container
            - name: BEYLA_OPEN_PORT
              value: "8443"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://grafana-alloy:4318"
              # required if you want kubernetes metadata decoration
            - name: BEYLA_KUBE_METADATA_ENABLE
              value: "true"

For more information about the different configuration options, check the Configuration section of this documentation site.

Deploy Beyla as a Daemonset

You can also deploy Beyla as a Daemonset. This is the preferred way if:

  • You want to instrument a Daemonset
  • You want to instrument multiple processes from a single Beyla instance, or even all of the processes in your cluster.

Using the previous example (the goblog pod), we cannot select the process to instrument by using its open port, because the port is internal to the Pod. At the same time multiple instances of the service would have different open ports. In this case, we will need to instrument by using the application service executable name (see later example).

In addition to the privilege requirements of the sidecar scenario, you will need to configure the auto-instrument pod template with the hostPID: true option enabled, so that it can access all the processes running on the same host.

yaml 
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: beyla
  labels:
    app: beyla
spec:
  selector:
    matchLabels:
      app: beyla
  template:
    metadata:
      labels:
        app: beyla
    spec:
      hostPID: true # Required to access the processes on the host
      serviceAccountName: beyla # required if you want kubernetes metadata decoration
      containers:
        - name: autoinstrument
          image: grafana/beyla:latest
          securityContext:
            privileged: true
          env:
            # Select the executable by its name instead of BEYLA_OPEN_PORT
            - name: BEYLA_EXECUTABLE_NAME
              value: "goblog"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://grafana-alloy:4318"
              # required if you want kubernetes metadata decoration
            - name: BEYLA_KUBE_METADATA_ENABLE
              value: "true"

Deploy Beyla unprivileged

In all of the examples so far, privileged:true or the SYS_ADMIN Linux capability was used in the Beyla deployment’s securityContext section. While this works in all circumstances, there are ways to deploy Beyla in Kubernetes with reduced privileges if your security configuration requires you to do so. Whether this is possible depends on the Kubernetes version you have and the underlying container runtime used (e.g. Containerd, CRI-O or Docker).

The following guide is based on tests performed mainly by running containerd with GKE, kubeadm, k3s, microk8s and kind.

To run Beyla unprivileged, you need to run a privileged init container which performs setup tasks which require elevated privileges. Then you need to replace the privileged:true setting with a set of Linux capabilities.

  • CAP_BPF is required to install most of the eBPF probes, because Beyla tracks system calls.
  • CAP_SYS_PTRACE is required so that Beyla is able to look into the processes namespaces and inspect the executables. Beyla doesn’t use ptrace, but for some of the operations it does require this capability.
  • CAP_NET_RAW is required for using installing socket filters, which are used as a fallback for kretprobes for HTTP requests.
  • CAP_CHECKPOINT_RESTORE is required to open ELF files.
  • CAP_DAC_READ_SEARCH is required to open ELF files.
  • CAP_PERFMON is required to load BPF programs, i.e. be able to perform perf_event_open().
  • CAP_SYS_RESOURCE is required only on kernels < 5.11 so that Beyla can increase the amount of locked memory available.

In addition to these Linux capabilities, many Kubernetes versions include AppArmour, which tough policies adds additional restrictions to unprivileged containers. By default, the AppArmour policy restricts the use of mount and the access to /sys/fs/ directories. Beyla uses the BPF Linux file system to store pinned BPF maps, for communication among the different BPF programs. For this reason, Beyla either needs to mount a BPF file system, or write to /sys/fs/bpf, which are both restricted.

Because of the AppArmour restriction, to run Beyla as unprivileged container, you need to either:

  • Set container.apparmor.security.beta.kubernetes.io/beyla: "unconfined" in your Kubernetes deployment files.
  • Set a modified AppArmour policy which allows Beyla to perform mount.

Note Since the beyla container does not have the privileges required to mount or un-mount the BPF filesystem, this sample leaves the BPF filesystem mounted on the host, even after the sample is deleted. This samples uses a unique path for each namespace to ensure re-use the same mount if Beyla is re-deployed, but to avoid collisions if multiple instances of Beyla is run in different namespaces.

Note Loading BPF programs requires that Beyla is able to read the Linux performance events, or at least be able to execute the Linux Kernel API perf_event_open(). This permission is granted by CAP_PERFMON or more liberally through CAP_SYS_ADMIN. Since both CAP_PERFMON and CAP_SYS_ADMIN grant Beyla the permission to read performance events, you should use CAP_PERFMON because it grants lesser permissions. However, at system level, the access to the performance events is controlled through the setting kernel.perf_event_paranoid, which you can read or write by using sysctl or by modifying the file /proc/sys/kernel/perf_event_paranoid. The default setting for kernel.perf_event_paranoid is typically 2, which is documented under the perf_event_paranoid section in the kernel documentation. Some Linux distributions define higher levels for kernel.perf_event_paranoid, for example Debian based distributions also use kernel.perf_event_paranoid=3, which disallows access to perf_event_open() without CAP_SYS_ADMIN. If you are running on a distribution with kernel.perf_event_paranoid setting higher than 2, you can either modify your configuration to lower it to 2 or use CAP_SYS_ADMIN instead of CAP_PERFMON.

An example of a Beyla unprivileged container configuration can be found below, or you can download the full example deployment file:

yaml 
...
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: beyla
  namespace: beyla-demo
  labels:
    k8s-app: beyla
spec:
  selector:
    matchLabels:
      k8s-app: beyla
  template:
    metadata:
      labels:
        k8s-app: beyla
      annotations:
        # We need to set beyla container as unconfined so it is able to write
        # the BPF file system.
        # Instead of 'unconfined', you can define a more refined policy which allows Beyla to use 'mount'
        container.apparmor.security.beta.kubernetes.io/beyla: "unconfined" # <-- Important
    spec:
      serviceAccount: beyla
      hostPID: true           # <-- Important. Required in Daemonset mode so Beyla can discover all monitored processes
      initContainers:
        - name: mount-bpf-fs
          image: grafana/beyla:latest
          args:
          # Create the directory and mount the BPF filesystem.
          - 'mkdir -p /sys/fs/bpf/$BEYLA_BPF_FS_PATH && mount -t bpf bpf /sys/fs/bpf/$BEYLA_BPF_FS_PATH'
          command:
          - /bin/bash
          - -c
          - --
          securityContext:
            # The init container is privileged so that it can use bidirectional mount propagation
            privileged: true
          volumeMounts:
          - name: bpffs
            mountPath: /sys/fs/bpf
            # Make sure the mount is propagated back to the host so it can be used by the Beyla container
            mountPropagation: Bidirectional
          env:
            - name: KUBE_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
              # Use a unique path for each namespace to prevent collisions with other namespaces.
            - name: BEYLA_BPF_FS_PATH
              value: beyla-$(KUBE_NAMESPACE)
      containers:
      - name: beyla
        terminationMessagePolicy: FallbackToLogsOnError
        image: grafana/beyla:latest
        env:
          - name: BEYLA_PRINT_TRACES
            value: "true"
          - name: BEYLA_KUBE_METADATA_ENABLE
            value: "autodetect"
          - name: KUBE_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
            # Use a unique path for each namespace to prevent collisions with other namespaces.
          - name: BEYLA_BPF_FS_PATH
            value: beyla-$(KUBE_NAMESPACE)
          - name: BEYLA_BPF_FS_BASE_DIR
            value: /sys/fs/bpf
          ...
        securityContext:
          runAsUser: 0
          readOnlyRootFilesystem: true
          capabilities:
            add:
              - BPF                 # <-- Important. Required for most eBPF probes to function correctly.
              - SYS_PTRACE          # <-- Important. Allows Beyla to access the container namespaces and inspect executables.
              - NET_RAW             # <-- Important. Allows Beyla to use socket filters for http requests.
              - CHECKPOINT_RESTORE  # <-- Important. Allows Beyla to open ELF files.
              - DAC_READ_SEARCH     # <-- Important. Allows Beyla to open ELF files.
              - PERFMON             # <-- Important. Allows Beyla to load BPF programs.
              #- SYS_RESOURCE       # <-- pre 5.11 only. Allows Beyla to increase the amount of locked memory.
              #- SYS_ADMIN          # <-- Required for Go application trace context propagation, or if kernel.perf_event_paranoid >= 3 on Debian distributions.
            drop:
              - ALL
        volumeMounts:
        - name: var-run-beyla
          mountPath: /var/run/beyla
        - name: cgroup
          mountPath: /sys/fs/cgroup
        - name: bpffs
          mountPath: /sys/fs/bpf
          mountPropagation: HostToContainer # <-- Important. Allows Beyla to see the BPF mount from the init container
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - name: var-run-beyla
        emptyDir: {}
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      - name: bpffs
        hostPath:
          path: /sys/fs/bpf
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-service
  namespace: beyla-demo
  ...
---

Providing an external configuration file

In the previous examples, Beyla was configured via environment variables. However, you can also configure it via an external YAML file (as documented in the Configuration section of this site).

To provide the configuration as a file, the recommended way is to deploy a ConfigMap with the intended configuration, then mount it into the Beyla Pod, and refer to it with the BEYLA_CONFIG_PATH environment variable.

Example of ConfigMap with the Beyla YAML documentation:

yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: beyla-config
data:
  beyla-config.yml: |
    print_traces: true
    grafana:
      otlp:
        submit: ["metrics","traces"]
    otel_traces_export:
      sampler:
        name: parentbased_traceidratio
        arg: "0.01"
    routes:
      patterns:
        - /factorial/{num}

Example of Beyla DaemonSet configuration, mounting and accessing to the previous ConfigMap:

yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: beyla
spec:
  selector:
    matchLabels:
      instrumentation: beyla
  template:
    metadata:
      labels:
        instrumentation: beyla
    spec:
      serviceAccountName: beyla
      hostPID: true #important!
      containers:
        - name: beyla
          image: grafana/beyla:latest
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            readOnlyRootFilesystem: true
          # mount the previous ConfigMap as a folder
          volumeMounts:
            - mountPath: /config
              name: beyla-config
            - mountPath: /var/run/beyla
              name: var-run-beyla
          env:
            # tell beyla where to find the configuration file
            - name: BEYLA_CONFIG_PATH
              value: "/config/beyla-config.yml"
      volumes:
        - name: beyla-config
          configMap:
            name: beyla-config
        - name: var-run-beyla
          emptyDir: {}

Providing secret configuration

The previous example is valid for regular configuration but should not be used to pass secret information like passwords or API keys.

To provide secret information, the recommended way is to deploy a Kubernetes Secret. For example, this secret contains some fictional Grafana Cloud credentials:

yaml 
apiVersion: v1
kind: Secret
metadata:
  name: grafana-secret
type: Opaque
stringData:
  grafana-user: "123456"
  grafana-api-key: "xxxxxxxxxxxxxxx"

Then you can access the secret values as environment variables. Following the previous DaemonSet example, this would be achieved by adding the following env section to the Beyla container:

yaml 
env:
  - name: GRAFANA_CLOUD_ZONE
    value: prod-eu-west-0
  - name: GRAFANA_CLOUD_INSTANCE_ID
    valueFrom:
      secretKeyRef:
        key: grafana-user
        name: grafana-secret
  - name: GRAFANA_CLOUD_API_KEY
    valueFrom:
      secretKeyRef:
        key: grafana-api-key
        name: grafana-secret