How to use Grafana Beyla in Grafana Alloy for eBPF-based auto-instrumentation

At GrafanaCON last month, we announced Grafana Alloy, our open source distribution of the OpenTelemetry Collector. Alloy is a telemetry collector that is 100% OTLP compatible and offers native pipelines for OpenTelemetry and Prometheus telemetry formats, supporting metrics, logs, traces, and profiles.

Today, we are excited to share that Grafana Beyla is now available in Grafana Alloy as the default eBPF-based application auto-instrumentation solution. With this, Alloy users can automatically capture metrics and traces of running services and connect to their existing telemetry pipelines. Grafana Alloy 1.1, released this month, comes with Grafana Beyla 1.5.

In this blog post, we’ll walk through some examples of how to use Grafana Beyla in Grafana Alloy, including how to get RED metrics from your running services and how to instrument your applications automatically in Kubernetes.

Get RED metrics from your running services

Let’s assume that you have a running service that accepts HTTP requests and listens to port 8080. Typically, you would need to add a Prometheus client library to your service and instrument your code to collect RED metrics. 

With the following configuration, you can get RED metrics from your service without changing your code:

alloy Copy

beyla.ebpf "default" {
    open_port = "8080"
}

prometheus.scrape "beyla" {
  targets = beyla.ebpf.default.targets
  forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
  endpoint {
    url = <PROMETHEUS_REMOTE_WRITE_URL>

    basic_auth {
      username = <USERNAME>
      password = <PASSWORD>
    }
  }
}

With this configuration, the beyla.ebpf component instruments the service that listens to port 8080. The open_port argument accepts a range of ports, a single port, or a comma-separated list of ports, so you can instrument multiple services at once. You can also specify the path to the executable of the service to be instrumented with the executable_name argument.

The prometheus.scrape component scrapes the metrics from the beyla.ebpf component and forwards them to the Prometheus remote write endpoint.

Replace <PROMETHEUS_REMOTE_WRITE_URL>, <USERNAME>, and <PASSWORD> with your Prometheus remote write endpoint URL, username, and password.

You need to run Alloy with sudo privileges to allow the eBPF-based instrumentation to work.

For further details on the configuration options, refer to our documentation for the Grafana Alloy beyla.ebpf component.

Instrument your applications automatically in Kubernetes

With the Grafana Beyla 1.2 release, we announced native Kubernetes support for Beyla. Now, with Grafana Alloy, you can use a Helm chart to deploy Beyla in your Kubernetes cluster and automatically instrument your applications.

To demonstrate how this works, let’s revisit an example from the Beyla 1.2 release post.

1. Prepare the Alloy environment in Kubernetes

You need to install the Helm chart for Grafana Alloy in your Kubernetes cluster.

bash Copy

helm install --namespace alloy alloy grafana/alloy

This command installs the Grafana Alloy Helm chart in the alloy namespace.

2. Deploy services

You can instrument any HTTP, HTTPS, or gRPC services in your Kubernetes cluster. You can use the sampleapps.yml file from Grafana Beyla’s repository and deploy it with the command kubectl apply -f sampleapps.yml.

3. Configure credentials

Alloy can export metrics and traces to any OpenTelemetry endpoint, as well as expose metrics as a Prometheus endpoint. However, we recommend using the Grafana Tempo remote write endpoint in Grafana Cloud. (Don’t already have a free Grafana Cloud account? Sign up for one today!) 

From the Grafana Cloud portal, find the Tempo box and click Send Traces.

Create a secrets.yml file with your Grafana Cloud credentials for Prometheus and Tempo remote write. Deploy it with the command kubectl apply -f secrets.yml.

yaml Copy

apiVersion: v1
kind: Secret
metadata:
  namespace: alloy
  name: grafana-credentials
type: Opaque
stringData:
  tempo-rw-user: "tempo-user"
  tempo-rw-pwd: "tempo-pwd"

4. Create a ConfigMap with Alloy configuration

Copy the following contents into a file (for example, config.alloy) and deploy it with the command kubectl create configmap --namespace alloy alloy-config "--from-file=config.alloy=./config.alloy".

alloy Copy

beyla.ebpf "default" {
    attributes {
   	 kubernetes {
   		 enable = "true"
   	 }
    }

    discovery {
   	 services {
   		 exe_path   = "http"
   		 open_ports = "80"
   	 }
    }

    output {
   	 traces = [otelcol.processor.batch.default.input]
    }
}


otelcol.processor.batch "default" {
    output {
   	traces  = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
    }
}

otelcol.exporter.otlp "grafana_cloud_tempo" {
    client {
   	 endpoint = "tempo-us-central1.grafana.net:443"
   	 auth 	= otelcol.auth.basic.grafana_cloud_tempo.handler
    }
}

otelcol.auth.basic "grafana_cloud_tempo" {
    username = env("TEMPO_REMOTE_WRITE_USERNAME")
    password = env("TEMPO_REMOTE_WRITE_PASSWORD")
}

In this configuration, Beyla is configured to instrument the services running in the Kubernetes cluster and send the traces to Grafana Cloud Traces.

The argument discovery > services > exe_path specifies the path to the executable of the services to be instrumented. The discovery > services > open_ports argument specifies the port where the services are listening.

Metrics and traces can be decorated with the metadata of the Kubernetes entities running the automatically instrumented services. To enable this feature, set the attributes > kubernetes > enable argument to true.

The output section of the Beyla component specifies that the traces should be sent to otelcol.exporter.otlp. The otelcol.exporter.otlp section configures the OTLP exporter to send the traces to Grafana Cloud Tempo.

5. Deploy Alloy with Helm

Create a values.yaml file with the configuration for the Alloy Helm chart. You can use the values.yaml file from Grafana Beyla’s GitHub repository as a reference, and then deploy it with the command helm upgrade --namespace alloy alloy grafana/alloy -f values.yaml.

Please note that:

• To run in DaemonSet mode, Beyla requires access to all the processes in the node. Therefore, hostPID: true is set in the controller section.
• The Beyla container runs with privileges, as it has to perform privileged actions such as loading BPF programs and creating BPF maps. Therefore privileged: true is set in the securityContext section. To run Beyla as an unprivileged container — meaning, without the privileged: true option — visit the Deploy Beyla unprivileged guide.
• The extraEnv section sets the environment variables for Tempo remote write credentials.

6. Test the setup

With the kubectl port-forward commands from the first step still running, test both web server instances. For example:

Copy

curl http://localhost:8080
curl http://localhost:8080/foo
curl http://localhost:8081
curl http://localhost:8081/foo

Now, go to the instance in Grafana Cloud, and from the Explore section in the left panel, select the data source for the traces (usually named grafanacloud-<your user name>-traces).

To search for all the traces, select the Search box in the query bar, leave the form empty, and click Run query.

This will show the traces for the docs instance (port 8081). You might see traces from your own services, but shouldn’t see traces from the website service, as it has not been instrumented by Beyla.

In the trace details, the resource attributes of the traces are decorated with the metadata of the Kubernetes Pod running the instrumented service.

What’s next?

In this blog post, we walked through how to use Grafana Beyla in Grafana Alloy to get RED metrics from your running services, as well as instrument your applications automatically in Kubernetes.

In the future, we plan to add more configuration options to Beyla to allow users to customize the instrumentation of their applications. We also plan to support more protocols to automatically instrument. Finally, in future versions of Beyla, we plan to add support for more eBPF-based features, such as network monitoring. Stay tuned for more updates!