Beyla tutorials on Grafana Labs

Beyla and Kubernetes walkthrough

Mon, 17 Mar 2025 08:44:01 +0100

Beyla and Kubernetes walkthrough

Kubernetes is fully integrated into the Beyla operation mode.

On one side, metrics and traces can be decorated with the metadata of the kubernetes entities running the automatically instrumented services.

On the other side, DaemonSet has become the preferred deployment mode for Beyla: thanks to the versatility of the new service selectors, a user can precisely define which services need to be instrumented and which don’t. A single instance of Beyla will be able to instrument the selected group of services within a single Kubernetes node.

Beyla service selectors

A service selector is a set of properties that let Beyla to query which processes need to be instrumented.

When Beyla is deployed as a regular operating system process that instrument other processes, the unique service selectors are the network port where the instrumented process should be listening to (can be specified with the BEYLA_OPEN_PORT environment variable) or a regular expression to match against the executable file name of the process to instrument (BEYLA_EXECUTABLE_NAME environment variable).

To select multiple groups of processes, the Beyla YAML configuration file format provides a discovery.services section that accepts multiple selector groups:

discovery:
  services:
    # Instrument any process using the ports from 8080 to 8089
    - open_ports: 8080-8089
    # Instrument any process whose executable contains "http"
    - exe_path: "http"
    # Instrument any process with an executable containing "nginx"
    # and using the port 443 (both conditions must be fulfilled)
    - open_ports: 443
      exe_path: "nginx"

The above criteria are insufficient for Kubernetes pods where the ports are ephemeral and internal to the pods. Also, pods are a level of abstraction that should hide details such as the name of their executables. For that reason, Beyla v1.2 introduces the new Kubernetes service selection criteria. All of them accept a Go RE2-syntax regular expression as value:

k8s_namespace: only instrument applications in the namespace matching the provided regular expression.
k8s_deployment_name: only instrument Pods that belong to a Deployment with a name matching the provided regular expression.
k8s_replicaset_name: only instrument Pods that belong to a ReplicaSet with a name matching the provided regular expression.
k8s_pod_name: only instrument Pods with a name matching the provided regular expression.

Example scenario

1. Deploy testing instrumentable services

You can instrument any HTTP or HTTPS service in your Kubernetes cluster. If you prefer, you can first try instrumenting the dummy services provided in this example.

The following Kubernetes example file contains two Apache HTTP servers: one pretends to be a company website and the other pretends to be a documentation site (docs). Let’s ignore that both servers will just return an “It Works!” string when the root directory is requested and a 404 error if any other path is requested.

Copy the following contents into a file (for example, sampleapps.yml) and deploy it with the command kubectl apply -f sampleapps.yml.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: docs
spec:
  replicas: 2
  selector:
    matchLabels:
      app: docs
  template:
    metadata:
      labels:
        app: docs
    spec:
      containers:
        - name: docs-server
          image: httpd:latest
          ports:
            - containerPort: 80
              protocol: TCP
              name: http
---
apiVersion: v1
kind: Service
metadata:
  name: docs
spec:
  selector:
    app: docs
  ports:
    - protocol: TCP
      port: 80
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: website
spec:
  replicas: 2
  selector:
    matchLabels:
      app: website
  template:
    metadata:
      labels:
        app: website
    spec:
      containers:
        - name: website-server
          image: httpd:latest
          ports:
            - containerPort: 80
              protocol: TCP
              name: http
---
apiVersion: v1
kind: Service
metadata:
  name: website
spec:
  selector:
    app: website
  ports:
    - protocol: TCP
      port: 80

To test that they are up and running, open two terminal sessions and run one of each command below on a different session:

# Redirect website to local port 8080
kubectl port-forward services/website 8080:80

# Redirect docs site to local port 8081
kubectl port-forward services/docs 8081:80

From your computer, each request to http://localhost:8080 will be a hypothetical request to the company website and each request to http://localhost:8081 will be a hypothetical request to the documentation website.

2. Create `beyla` namespace

Before configuring and deploying Beyla, let’s create a beyla namespace. We will group there all the permissions, configurations and deployments related to it:

kubectl create namespace beyla

3. Get Grafana Cloud credentials

Beyla can export metrics and traces to any OpenTelemetry endpoint, as well as exposing metrics as a Prometheus endpoint. However, we recommend using the OpenTelemetry endpoint in Grafana Cloud. You can get a Free Grafana Cloud Account at Grafana’s website.

From the Grafana Cloud Portal, look for the OpenTelemetry box and click Configure.

Under Password / API token click Generate now and follow the instructions to create a default API token.

The Environment Variables will be populated with a set of standard OpenTelemetry environment variables which will provide the connection endpoint and credentials information for Beyla.

From the Environment Variables section, copy the OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS values and create a new secret from them. For example, create the following secret file and apply it:

apiVersion: v1
kind: Secret
metadata:
  namespace: beyla
  name: grafana-credentials
type: Opaque
stringData:
  otlp-endpoint: "https://otlp-gateway-prod-eu-west-0.grafana.net/otlp"
  otlp-headers: "Authorization=Basic ...rest of the secret header value..."

3. Configure and run Beyla

Next, you need to provide Beyla with permissions to watch and inspect the metadata of the diverse Kubernetes resources that Beyla’s discovery mechanism requires. You must create the following YAML file and apply it:

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: beyla
  name: beyla
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: beyla
rules:
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: beyla
subjects:
  - kind: ServiceAccount
    name: beyla
    namespace: beyla
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: beyla

And now, deploy Beyla by creating the following Kubernetes entities:

A ConfigMap storing the beyla-config.yml Beyla configuration file, which defines the service discovery criteria. To verify that Beyla is able to discriminate by service instance even if they run the same image and executable, Beyla is configured to select ONLY the docs Apache web server.
A Beyla DaemonSet providing the Beyla pod and its configuration:
- Loads the beyla-config.yml file from the ConfigMap, as specified in the BEYLA_CONFIG_PATH environment variable.
- References to the grafana-secrets values for the endpoint and credentials.
- Uses the beyla ServiceAccount to get all the permissions.

Copy and deploy the following YAML file:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: beyla
  name: beyla-config
data:
  beyla-config.yml: |
    # this is required to enable kubernetes discovery and metadata
    attributes:
      kubernetes:
        enable: true
    # this will provide automatic routes report while minimizing cardinality
    routes:
      unmatched: heuristic
    # let's instrument only the docs server
    discovery:
      services:
        - k8s_deployment_name: "^docs$"
        # uncomment the following line to also instrument the website server
        # - k8s_deployment_name: "^website$"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: beyla
  name: beyla
spec:
  selector:
    matchLabels:
      instrumentation: beyla
  template:
    metadata:
      labels:
        instrumentation: beyla
    spec:
      serviceAccountName: beyla
      hostPID: true # mandatory!
      containers:
        - name: beyla
          image: grafana/beyla:1.2
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true # mandatory!
            readOnlyRootFilesystem: true
          volumeMounts:
            - mountPath: /config
              name: beyla-config
            - mountPath: /var/run/beyla
              name: var-run-beyla
          env:
            - name: BEYLA_CONFIG_PATH
              value: "/config/beyla-config.yml"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: grafana-credentials
                  key: otlp-endpoint
            - name: OTEL_EXPORTER_OTLP_HEADERS
              valueFrom:
                secretKeyRef:
                  name: grafana-credentials
                  key: otlp-headers
      volumes:
        - name: beyla-config
          configMap:
            name: beyla-config
        - name: var-run-beyla
          emptyDir: {}

Also notice:

To run in DaemonSet mode, Beyla requires to have access to all the processes in the node. Then the Beyla Pod requires to run with hostPID: true.
The Beyla container needs to run with privileged: true, as it requires to perform privileged actions such as loading BPF programs and creating BPF maps. For running Beyla as unprivileged container, i.e. without the privileged: true option, visit the Deploy Beyla unprivileged guide.

4. Test your instrumented services and see the results in Grafana

With the kubectl port-forward commands from the firs step still running, test both web server instances. For example:

curl http://localhost:8080
curl http://localhost:8080/foo
curl http://localhost:8081
curl http://localhost:8081/foo

Some requests will return 404 error, but it’s OK because they are also instrumented.

Now, go to the instance in Grafana Cloud, and from the Explore section in the left panel, select the data source for the traces (usually named grafanacloud-<your user name>-traces).

To search for all the traces, select the Search box in the Query bar, leave the form empty, and click Run query:

This will show the traces for the docs instance (port 8081). You might see traces from your own services, but shouldn’t see traces from the website service, as it has not been instrumented by Beyla.

In the trace details, the resource attributes of the traces are decorated with the metadata of the Kubernetes Pod running the instrumented service:

Getting started with Beyla

Mon, 17 Mar 2025 08:44:01 +0100

Getting started with Beyla

To reduce the time it takes to instrument an application and improve the adoption of Application Observability, Grafana built Beyla, an eBPF auto-instrumentation tool, that is able to report transactions span information, as well as RED metrics for Linux HTTP/S and gRPC services, without any application code or configuration changes.

eBPF overview

eBPF stands for Extended Berkeley Packet Filter, and allows attaching applications to different points of the Linux Kernel. eBPF applications run in privileged mode and allow the runtime information of the Linux Kernel to be inspected: system calls, network stack, as well as inserting probes in user space applications.

The eBPF applications are safe, they are compiled for their own Virtual Machine instruction set and run in a sandboxed environment that verifies each loaded eBPF program for memory access safety and finite execution time. Unlike older technologies, such as the natively-compiled Kprobes and Uprobes, there is no chance that a poorly programmed probe will cause the Linux Kernel to hang.

After being the eBPF binaries have been verified they are compiled with a Just-In-Time (JIT) compiler for the native host architecture (x86-64, ARM64, etc). This allows for efficient and fast execution.

The eBPF code is loaded from ordinary applications running in user space. The kernel and the user space applications can share information through a set of well defined communication mechanisms, which are provided by the eBPF specification. For example: ring buffers, arrays, hash maps, etc.

Running an instrumented service

For this quick start tutorial, instrument any HTTP, HTTPS or gRPC Go service that uses any of the following libraries:

Standard net/http
Gorilla Mux
Gin
gRPC-Go

HTTP and HTTPS services written in other languages can also be instrumented:

Node.js (HTTP 1.1 and HTTPS with OpenSSL)
Python (HTTP 1.1 and HTTPS with OpenSSL)
Rust (HTTP 1.1 and HTTPS with OpenSSL)
Ruby (HTTP 1.1 and HTTPS with OpenSSL)
.NET Core 6+ (HTTP 1.1 and HTTPS with OpenSSL)
Java (HTTP 1.1)

The HTTP 1.1 and OpenSSL support is generic, services written in different programming languages than those listed above might work, but haven’t been tested.

If you don’t have a service to instrument, create a server.go file with the following code:

package main

import (
	"net/http"
	"strconv"
	"time"
)

func handleRequest(rw http.ResponseWriter, req *http.Request) {
	status := 200
	for k, v := range req.URL.Query() {
		if len(v) == 0 {
			continue
		}
		switch k {
		case "status":
			if s, err := strconv.Atoi(v[0]); err == nil {
				status = s
			}
		case "delay":
			if d, err := time.ParseDuration(v[0]); err == nil {
				time.Sleep(d)
			}
		}
	}
	rw.WriteHeader(status)
}

func main() {
	http.ListenAndServe(":8080", http.HandlerFunc(handleRequest))
}

The code implements an HTTP service that accepts request on port 8080. The HTTP handler behavior can be specified with the following query parameters:

status will override the returned HTTP status code (which defaults to 200). For example curl -v "http://localhost:8080/foo?status=404" will return a 404 status code.
delay will artificially increase the service response time. For example curl "http://localhost:8080/bar?delay=3s" will take at least 3 seconds to complete.

Download the server.go file from this tutorial.

Run the test HTTP service with the following command:

go run server.go

Instrument a service

Set up Beyla as a standalone linux process by following the standalone setup documentation.

First, we will locally check that Beyla is able to instrument the provided test server application, after configuring it to print the traces to the standard output.

Set environment variables and run Beyla:

BEYLA_PRINT_TRACES=true BEYLA_OPEN_PORT=8080 sudo -E beyla

The BEYLA_PRINT_TRACES=true configuration option tells Beyla to log any trace to the standard output. The BEYLA_OPEN_PORT=8080 option tells Beyla to instrument the service that owns the port 8080. Since Beyla requires administrator rights to load eBPF programs, the beyla command must run with sudo -E (or as a root user).

After running the above command, you should see a log message confirming that the test service has been found:

time=2023-11-14T09:10:00.513Z level=INFO msg="instrumenting process"
component=discover.TraceAttacher cmd=/tmp/go-build898688565/b001/exe/server pid=8710

Now, open a new terminal and send a few HTTP GET calls to the test service:

curl -v "http://localhost:8080/hello"
curl -v "http://localhost:8080/bye"

For each request, Beyla will log trace information to the first terminal:

2023-11-14 09:11:13 (2.89ms[859.99µs]) 200 GET /hello
  [127.0.0.1]->[localhost:8080] size:0B svc=[{server go your-hostname-8710}]

2023-11-14 09:11:13 (1.87ms[191µs]) 200 GET /bye
  [127.0.0.1]->[localhost:8080] size:0B svc=[{server go your-hostname-8710}]

The output format is:

Request_time (response_duration) status_code http_method path
  source->destination request_size service_id

Experiment with the curl command and make additional requests to see how it affects the trace output. For example, the following request would send a 6-bytes POST request and the service will take 200ms to respond:

curl -X POST -d "abcdef" "http://localhost:8080/post?delay=200ms"

Beyla will log the following trace information:

2023-11-14 09:12:49 (208.32ms[206.79ms]) 200 POST /post
[127.0.0.1]->[localhost:8080] size:6B svc=[{server go your-hostname-8710}]

Send data to Grafana Cloud

Once we have verified that our application is correctly instrumented, we can set up a Grafana Cloud OpenTelemetry exporter to read the generated traces and forward them to Grafana Cloud. You can get a Free Grafana Cloud Account at Grafana’s website.

For information on how to configure Beyla to submit data to other OpenTelemetry collectors, or how to generate Prometheus metrics, see the configuration options documentation.

There are two ways to forward your OpenTelemetry traces to Grafana Cloud:

Using Grafana Alloy and configuring Beyla to forward the traces to it via the standard OpenTelemetry export.
Configuring Beyla to submit data directly to the Grafana Cloud OpenTelemetry Protocol endpoint, as shown in this tutorial.

Running Grafana Beyla with your Grafana Credentials

In your Grafana Cloud Portal, click on the “Details” button in the “OpenTelemetry” box. Next, copy your Grafana OTLP Endpoint and Instance ID, as in the image below.

Also generate a Password/API token with metrics push privileges.

Now you can run Beyla by using the above information to set the OTEL_EXPORTER_OTLP_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID and GRAFANA_CLOUD_API_KEY environment variables.

The GRAFANA_CLOUD_SUBMIT environment variable (whose value defaults to traces) lets you choose which type of data to submit to the Grafana OpenTelemetry endpoint: metrics and/or traces. To make use of the metrics dashboard presented in the next section, we will set GRAFANA_CLOUD_SUBMIT=metrics.

For example:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-eu-west-0.grafana.net/otlp
export GRAFANA_CLOUD_SUBMIT=metrics
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY="your api key here..."

BEYLA_OPEN_PORT=8080 sudo -E beyla

Optionally, open another terminal and run the following command to generate some artificial load:

while true; do curl -v "http://localhost:8080/service?delay=1s"; done

To verify that metrics are properly received by Grafana, you can go to the left panel, choose the Explore tab and your Prometheus data source. Next, write http_ in the Metrics Browser input field and you should see the available metric names in the auto-complete drop-down.

Add the Beyla RED Metrics Dashboard

You could start composing your PromQL queries for better visualization of your auto-instrumented RED metrics; to save you time, we provide a sample public dashboard with some basic information.

To import the sample dashboard into your Grafana instance, choose “Dashboards” in the Grafana left panel. Next, in the Dashboards page, click on the “New” drop-down menu and select “Import”:

In the “Import via grafana.com” textbox, copy the Grafana ID from the Beyla RED Metrics dashboard: 19923.

Rename the dashboard to match your service, select the folder and, most importantly, select the data source in the prometheus-data-source drop-down at the bottom.

And voilà! you can see some of your test RED metrics:

The dashboard contains the following components:

A list with the slowest HTTP routes for all instrumented services. Since you only have a single service, only one entry appears. If you configure Beyla to report the HTTP routes, many entries could appear there, one for each HTTP path seen by the server.
A list with the slowest GRPC methods. Since the test service in this tutorial only serves HTTP, this table is empty.
For each instrumented service, a list of RED metrics for the inbound (server) traffic. This includes:
- Duration: average and top percentiles for both HTTP and gRPC traffic.
- Request rate: number of requests per second, faceted by its HTTP or gRPC return code.
- Error rate as a percentage of 5xx HTTP responses or non-zero gRPC responses over the total of the requests. They are faceted by return code.
For each instrumented service, a list of RED metrics for the outbound (client) traffic. In the above screenshot they are empty because the test service does not perform HTTP or gRPC calls to other services.
- The Duration, Request Rate and Errors charts are analogues to the inbound traffic charts, with the only difference that 4xx return codes are also considered errors on the client side.

At the top of the chart, you can use the “Service” dropdown to filter the services you want to visualize.

Conclusions and future work

eBPF proved to be a low-overhead, safe, and reliable way to observe some basic metrics for HTTP/gRPC services. Beyla is not a replacement for language specific agents, however it significantly decreases the landing time of your application insights in Grafana. Beyla does not require any code changes, recompilation nor repackaging, simply run it together with your service, and your application metrics will start to flow.

eBPF also allows you to get deeper insights which manual instrumentation doesn’t. For example, Beyla is able to show you how much time a request is enqueued, after the connection is established, and before its code is actually executed (requires exporting OpenTelemetry traces, but this functionality is not explained in this tutorial).

Beyla has its limitations too. It only provides generic metrics and transaction level trace span information. Language agents and manual instrumentation is still recommended, so that you can specify the granularity of each part of the code to be instrumented, putting the focus on your critical operations.

Another limitation to consider is that Beyla requires elevated privileges; not actually a root user, but at least it has to run with the CAP_SYS_ADMIN capability. If you run the tool as a container (Docker, Kubernetes…), it has to be privileged.

In the future, we plan to add metrics about other well-established protocols, like database or message queuing connections.

Distributed tracing is only supported for Go services, while other programming language support remains on the road-map. Distributed tracing can correlate requests from multiple services (web, database, messaging…). One complexity of distributed tracing is the injection of client-side headers and matching them to the context of the server-side requests. Each release is making progressive advances towards this goal.

Another shorter term goal is to reduce the surface of the code that requires administrative privileges, executing a small eBPF loader with root or CAP_SYS_ADMIN privileges, and running the rest of data processing/exposition with normal user privileges.

Run Beyla in Kubernetes using Grafana Alloy Helm's chart

Mon, 17 Mar 2025 08:44:01 +0100

Run Beyla in Kubernetes using Grafana Alloy Helm’s chart

Grafana Alloy is a vendor-neutral distribution of the OpenTelemetry Collector. Alloy offers native pipelines for OpenTelemetry, Prometheus, and other telemetry signals.

Grafana Alloy bundles Beyla allowing you to instrument your applications at the same time you instrument your infrastructure. It also provides a Helm chart to deploy Alloy in Kubernetes.

In this tutorial, you learn how to deploy Beyla in Kubernetes using Grafana Alloy Helm’s chart. It’s based on the examples from the Beyla and Kubernetes walkthrough.

Prerequisites

A Kubernetes cluster, you can use kind to create a local cluster
kubectl installed and configured for your cluster
Helm installed
A Grafana Cloud account or a compatible Prometheus and/or OpenTelemetry backend to receive the data

1. Prepare the Alloy environment in Kubernetes

You need to install the Helm chart for Grafana Alloy in your Kubernetes cluster.

helm install --namespace alloy alloy grafana/alloy

This command installs the Grafana Alloy Helm chart in the alloy namespace.

2. Deploy services

You can instrument any HTTP or HTTPS service in your Kubernetes cluster.

Copy the following contents into a file, for example sampleapps.yml, and deploy it with the command kubectl apply -f sampleapps.yml.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: docs
spec:
  replicas: 2
  selector:
    matchLabels:
      app: docs
  template:
    metadata:
      labels:
        app: docs
    spec:
      containers:
        - name: docs-server
          image: httpd:latest
          ports:
            - containerPort: 80
              protocol: TCP
              name: http
---
apiVersion: v1
kind: Service
metadata:
  name: docs
spec:
  selector:
    app: docs
  ports:
    - protocol: TCP
      port: 80
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: website
spec:
  replicas: 2
  selector:
    matchLabels:
      app: website
  template:
    metadata:
      labels:
        app: website
    spec:
      containers:
        - name: website-server
          image: httpd:latest
          ports:
            - containerPort: 80
              protocol: TCP
              name: http
---
apiVersion: v1
kind: Service
metadata:
  name: website
spec:
  selector:
    app: website
  ports:
    - protocol: TCP
      port: 80

3. Configure credentials

Alloy can export metrics and traces to any OpenTelemetry endpoint, as well as exposing metrics as a Prometheus endpoint. However, it’s recommend using the Prometheus and Tempo remote write endpoints in Grafana Cloud. You can get a free Grafana Cloud Account.

From the Grafana Cloud Portal, look for the Prometheus box and click Send Metrics. For the Tempo box, click Send Traces.

Create a secrets.yml file with your Grafana Cloud credentials for Prometheus and Tempo remote write. Deploy it with the command kubectl apply -f secrets.yml.

apiVersion: v1
kind: Secret
metadata:
  namespace: alloy
  name: grafana-credentials
type: Opaque
stringData:
  prometheus-rw-user: "prom-user"
  prometheus-rw-pwd: "prom-pwd"
  tempo-rw-user: "tempo-user"
  tempo-rw-pwd: "tempo-pwd"

3. Create a ConfigMap with Alloy configuration

Create a ConfigMap with the Alloy configuration. Copy the following contents into a file, for example config.alloy:

beyla.ebpf "default" {
	attributes {
		kubernetes {
			enable = "true"
		}
	}

	discovery {
		services {
			exe_path   = "http"
			open_ports = "80"
		}
	}

	output {
		traces = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
	}
}

otelcol.exporter.otlp "grafana_cloud_tempo" {
	client {
		endpoint = "tempo-us-central1.grafana.net:443"
		auth     = otelcol.auth.basic.grafana_cloud_tempo.handler
	}
}

otelcol.auth.basic "grafana_cloud_tempo" {
	username = env("TEMPO_REMOTE_WRITE_USERNAME")
	password = env("TEMPO_REMOTE_WRITE_PASSWORD")
}

Deploy the configuration with the command:

kubectl create configmap --namespace alloy alloy-config "--from-file=config.alloy=./config.alloy"

With this configuration Beyla instruments the services running in the Kubernetes cluster and send traces to Grafana Cloud Tempo and metrics to Prometheus.

The argument discovery > services > exe_path specifies the path to the executable of the services to instrument. The discovery > services > open_ports argument specifies the port where the services are listening.

The attributes > kubernetes > enable enables Kubernetes decoration for metrics and traces, which adds the metadata of the Kubernetes entities running the automatically instrumented services.

The prometheus.scrape section configures the Prometheus scrape configuration to collect the metrics from Beyla. The prometheus.remote_write section configures the remote write to send the metrics to Grafana Cloud Prometheus.

The output section configures that Beyla component sends traces to otelcol.exporter.otlp component. The otelcol.exporter.otlp section configures the OTLP exporter to send the traces to Grafana Cloud Tempo.

For further details on the configuration options, refer to the documentation of the Grafana Alloy Beyla component

4. Deploy Alloy with Helm

Create a values.yaml with the configuration for the Alloy Helm chart. Copy the following contents into a file, for example values.yaml.

# -- Overrides the chart's name. Used to change the infix in the resource names.
nameOverride: null

# -- Overrides the chart's computed fullname. Used to change the full prefix of
# resource names.
fullnameOverride: null

## Global properties for image pulling override the values defined under `image.registry` and `configReloader.image.registry`.
## If you want to override only one image registry, use the specific fields but if you want to override them all, use `global.image.registry`
global:
  image:
    # -- Global image registry to use if it needs to be overriden for some specific use cases (e.g local registries, custom images, ...)
    registry: ""

    # -- Optional set of global image pull secrets.
    pullSecrets: []

  # -- Security context to apply to the Grafana Alloy pod.
  podSecurityContext: {}

crds:
  # -- Whether to install CRDs for monitoring.
  create: true

## Various Alloy settings. For backwards compatibility with the grafana-agent
## chart, this field may also be called "agent". Naming this field "agent" is
## deprecated and will be removed in a future release.
alloy:
  configMap:
    # -- Create a new ConfigMap for the config file.
    create: false
    # -- Name of existing ConfigMap to use. Used when create is false.
    name: alloy-config
    # -- Key in ConfigMap to get config from.
    key: config.alloy

  clustering:
    # -- Deploy Alloy in a cluster to allow for load distribution.
    enabled: false

  # -- Minimum stability level of components and behavior to enable. Must be
  # one of "experimental", "public-preview", or "generally-available".
  stabilityLevel: "public-preview"

  # -- Path to where Grafana Alloy stores data (for example, the Write-Ahead Log).
  # By default, data is lost between reboots.
  storagePath: /tmp/alloy

  # -- Address to listen for traffic on. 0.0.0.0 exposes the UI to other
  # containers.
  listenAddr: 0.0.0.0

  # -- Port to listen for traffic on.
  listenPort: 12345

  # -- Scheme is needed for readiness probes. If enabling tls in your configs, set to "HTTPS"
  listenScheme: HTTP

  # --  Base path where the UI is exposed.
  uiPathPrefix: /

  # -- Enables sending Grafana Labs anonymous usage stats to help improve Grafana
  # Alloy.
  enableReporting: true

  # -- Extra environment variables to pass to the Alloy container.
  extraEnv:
  - name: PROMETHEUS_REMOTE_WRITE_USERNAME
    valueFrom:
      secretKeyRef:
        name: grafana-credentials
        key: prometheus-rw-user
  - name: PROMETHEUS_REMOTE_WRITE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: grafana-credentials
        key: prometheus-rw-pwd
  - name: TEMPO_REMOTE_WRITE_USERNAME
    valueFrom:
      secretKeyRef:
        name: grafana-credentials
        key: tempo-rw-user
  - name: TEMPO_REMOTE_WRITE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: grafana-credentials
        key: tempo-rw-pwd

  # -- Maps all the keys on a ConfigMap or Secret as environment variables. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#envfromsource-v1-core
  envFrom: []

  # -- Extra args to pass to `alloy run`: https://grafana.com/docs/alloy/latest/reference/cli/run/
  extraArgs: []

  # -- Extra ports to expose on the Alloy container.
  extraPorts: []
  # - name: "faro"
  #   port: 12347
  #   targetPort: 12347
  #   protocol: "TCP"

  mounts:
    # -- Mount /var/log from the host into the container for log collection.
    varlog: false
    # -- Mount /var/lib/docker/containers from the host into the container for log
    # collection.
    dockercontainers: false

    # -- Extra volume mounts to add into the Grafana Alloy container. Does not
    # affect the watch container.
    extra: []

  # -- Security context to apply to the Grafana Alloy container.
  securityContext:
    privileged: true # important!

  # -- Resource requests and limits to apply to the Grafana Alloy container.
  resources: {}

image:
  # -- Grafana Alloy image registry (defaults to docker.io)
  registry: "docker.io"
  # -- Grafana Alloy image repository.
  repository: grafana/alloy
  # -- (string) Grafana Alloy image tag. When empty, the Chart's appVersion is
  # used.
  tag: null
  # -- Grafana Alloy image's SHA256 digest (either in format "sha256:XYZ" or "XYZ"). When set, will override `image.tag`.
  digest: null
  # -- Grafana Alloy image pull policy.
  pullPolicy: IfNotPresent
  # -- Optional set of image pull secrets.
  pullSecrets: []

rbac:
  # -- Whether to create RBAC resources for Alloy.
  create: true

serviceAccount:
  # -- Whether to create a service account for the Grafana Alloy deployment.
  create: true
  # -- Additional labels to add to the created service account.
  additionalLabels: {}
  # -- Annotations to add to the created service account.
  annotations: {}
  # -- The name of the existing service account to use when
  # serviceAccount.create is false.
  name: null

# Options for the extra controller used for config reloading.
configReloader:
  # -- Enables automatically reloading when the Alloy config changes.
  enabled: true
  image:
    # -- Config reloader image registry (defaults to docker.io)
    registry: "ghcr.io"
    # -- Repository to get config reloader image from.
    repository: jimmidyson/configmap-reload
    # -- Tag of image to use for config reloading.
    tag: v0.12.0
    # -- SHA256 digest of image to use for config reloading (either in format "sha256:XYZ" or "XYZ"). When set, will override `configReloader.image.tag`
    digest: ""
  # -- Override the args passed to the container.
  customArgs: []
  # -- Resource requests and limits to apply to the config reloader container.
  resources:
    requests:
      cpu: "1m"
      memory: "5Mi"
  # -- Security context to apply to the Grafana configReloader container.
  securityContext: {}

controller:
  # -- Type of controller to use for deploying Grafana Alloy in the cluster.
  # Must be one of 'daemonset', 'deployment', or 'statefulset'.
  type: 'daemonset'

  # -- Number of pods to deploy. Ignored when controller.type is 'daemonset'.
  replicas: 1

  # -- Annotations to add to controller.
  extraAnnotations: {}

  # -- Whether to deploy pods in parallel. Only used when controller.type is
  # 'statefulset'.
  parallelRollout: true

  # -- Configures Pods to use the host network. When set to true, the ports that will be used must be specified.
  hostNetwork: false

  # -- Configures Pods to use the host PID namespace.
  hostPID: true # important!

  # -- Configures the DNS policy for the pod. https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
  dnsPolicy: ClusterFirst

  # -- Update strategy for updating deployed Pods.
  updateStrategy: {}

  # -- nodeSelector to apply to Grafana Alloy pods.
  nodeSelector: {}

  # -- Tolerations to apply to Grafana Alloy pods.
  tolerations: []

  # -- Topology Spread Constraints to apply to Grafana Alloy pods.
  topologySpreadConstraints: []

  # -- priorityClassName to apply to Grafana Alloy pods.
  priorityClassName: ''

  # -- Extra pod annotations to add.
  podAnnotations: {}

  # -- Extra pod labels to add.
  podLabels: {}

  # -- Whether to enable automatic deletion of stale PVCs due to a scale down operation, when controller.type is 'statefulset'.
  enableStatefulSetAutoDeletePVC: false

  autoscaling:
    # -- Creates a HorizontalPodAutoscaler for controller type deployment.
    enabled: false
    # -- The lower limit for the number of replicas to which the autoscaler can scale down.
    minReplicas: 1
    # -- The upper limit for the number of replicas to which the autoscaler can scale up.
    maxReplicas: 5
    # -- Average CPU utilization across all relevant pods, a percentage of the requested value of the resource for the pods. Setting `targetCPUUtilizationPercentage` to 0 will disable CPU scaling.
    targetCPUUtilizationPercentage: 0
    # -- Average Memory utilization across all relevant pods, a percentage of the requested value of the resource for the pods. Setting `targetMemoryUtilizationPercentage` to 0 will disable Memory scaling.
    targetMemoryUtilizationPercentage: 80

    scaleDown:
      # -- List of policies to determine the scale-down behavior.
      policies: []
        # - type: Pods
        #   value: 4
        #   periodSeconds: 60
      # -- Determines which of the provided scaling-down policies to apply if multiple are specified.
      selectPolicy: Max
      # -- The duration that the autoscaling mechanism should look back on to make decisions about scaling down.
      stabilizationWindowSeconds: 300

    scaleUp:
      # -- List of policies to determine the scale-up behavior.
      policies: []
        # - type: Pods
        #   value: 4
        #   periodSeconds: 60
      # -- Determines which of the provided scaling-up policies to apply if multiple are specified.
      selectPolicy: Max
      # -- The duration that the autoscaling mechanism should look back on to make decisions about scaling up.
      stabilizationWindowSeconds: 0

  # -- Affinity configuration for pods.
  affinity: {}

  volumes:
    # -- Extra volumes to add to the Grafana Alloy pod.
    extra: []

  # -- volumeClaimTemplates to add when controller.type is 'statefulset'.
  volumeClaimTemplates: []

  ## -- Additional init containers to run.
  ## ref: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
  ##
  initContainers: []

  # -- Additional containers to run alongside the Alloy container and initContainers.
  extraContainers: []

service:
  # -- Creates a Service for the controller's pods.
  enabled: true
  # -- Service type
  type: ClusterIP
  # -- NodePort port. Only takes effect when `service.type: NodePort`
  nodePort: 31128
  # -- Cluster IP, can be set to None, empty "" or an IP address
  clusterIP: ''
  # -- Value for internal traffic policy. 'Cluster' or 'Local'
  internalTrafficPolicy: Cluster
  annotations: {}
    # cloud.google.com/load-balancer-type: Internal

serviceMonitor:
  enabled: false
  # -- Additional labels for the service monitor.
  additionalLabels: {}
  # -- Scrape interval. If not set, the Prometheus default scrape interval is used.
  interval: ""
  # -- MetricRelabelConfigs to apply to samples after scraping, but before ingestion.
  # ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig
  metricRelabelings: []
  # - action: keep
  #   regex: 'kube_(daemonset|deployment|pod|namespace|node|statefulset).+'
  #   sourceLabels: [__name__]

  # -- Customize tls parameters for the service monitor
  tlsConfig: {}

  # -- RelabelConfigs to apply to samples before scraping
  # ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig
  relabelings: []
  # - sourceLabels: [__meta_kubernetes_pod_node_name]
  #   separator: ;
  #   regex: ^(.*)$
  #   targetLabel: nodename
  #   replacement: $1
  #   action: replace
ingress:
  # -- Enables ingress for Alloy (Faro port)
  enabled: false
  # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  # ingressClassName: nginx
  # Values can be templated
  annotations:
    {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  labels: {}
  path: /
  faroPort: 12347

  # pathType is only for k8s >= 1.1=
  pathType: Prefix

  hosts:
    - chart-example.local
  ## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
  extraPaths: []
  # - path: /*
  #   backend:
  #     serviceName: ssl-redirect
  #     servicePort: use-annotation
  ## Or for k8s > 1.19
  # - path: /*
  #   pathType: Prefix
  #   backend:
  #     service:
  #       name: ssl-redirect
  #       port:
  #         name: use-annotation

  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

Deploy the configuration with the command:

helm upgrade --namespace alloy alloy grafana/alloy -f values.yaml

To run in DaemonSet mode, Beyla requires to have access to all the processes in the node. Therefore set hostPID: true the controller section.
The Beyla container needs to run with privileges as it requires to perform privileged actions such as loading BPF programs and creating BPF maps. Therefore set privileged: true in securityContext section. For running Beyla as unprivileged container, that’s without the privileged: true option, visit the Deploy Beyla unprivileged guide.
The extraEnv section sets the environment variables for the Prometheus and Tempo remote write credentials.

5. Test the setup

With the kubectl port-forward commands from the first step still running, test both web server instances. For example:

curl http://localhost:8080
curl http://localhost:8080/foo
curl http://localhost:8081
curl http://localhost:8081/foo

Navigate to the instance in Grafana Cloud, and from the Explore section in the left panel, select the data source for the traces, named grafanacloud-<your user name>-traces.

To search for all the traces, select the Search box in the Query bar, leave the form empty, and click Run query:

This shows the traces for the docs instance on port 8081. You might see traces from your own services, but you shouldn’t see traces from the website service, as it Beyla isn’t instrumenting it.

In the trace details, the resource attributes of the traces have metadata of the Kubernetes Pod running the instrumented service:

Beyla tutorials on Grafana Labs

Beyla and Kubernetes walkthrough

Beyla and Kubernetes walkthrough

Beyla service selectors

Example scenario

1. Deploy testing instrumentable services

2. Create beyla namespace

3. Get Grafana Cloud credentials

3. Configure and run Beyla

4. Test your instrumented services and see the results in Grafana

Links

Getting started with Beyla

Getting started with Beyla

eBPF overview

Running an instrumented service

Instrument a service

Send data to Grafana Cloud

Running Grafana Beyla with your Grafana Credentials

Add the Beyla RED Metrics Dashboard

Conclusions and future work

Run Beyla in Kubernetes using Grafana Alloy Helm's chart

Run Beyla in Kubernetes using Grafana Alloy Helm’s chart

Prerequisites

1. Prepare the Alloy environment in Kubernetes

2. Deploy services

3. Configure credentials

3. Create a ConfigMap with Alloy configuration

4. Deploy Alloy with Helm

5. Test the setup

2. Create `beyla` namespace