Set up profiling with eBPF with Grafana Alloy on Grafana Labs

Setup eBPF Profiling on Kubernetes

Wed, 08 Apr 2026 14:38:28 +0000

Setup eBPF Profiling on Kubernetes

To set up eBPF profiling with Grafana Alloy on Kubernetes, you need to:

Verify that your cluster meets the prerequisites.
Add the Grafana helm repository.
Create an Alloy configuration file. For more information, refer to Configuration reference.
Install Alloy, refer to the installation instructions
Verify that profiles are received.

Before you begin

Before you begin, you need:

Helm and kubectl installed with access to your Kubernetes cluster.
A Pyroscope server where Alloy can send profiling data.
Access to Grafana with the Grafana Pyroscope data source provisioned.

Note
If you don’t have a Grafana or a Pyroscope server, you can use the [Grafana Cloud][gcloud] free plan to get started.

Verify that your cluster meets the requirements

The eBPF profiler requires a Linux kernel version >= 4.9 (due to BPF_PROG_TYPE_PERF_EVENT).

BPF_PROG_TYPE_PERF_EVENT is a type of eBPF program that can be attached to hardware or software events, such as performance monitoring counters or tracepoints, in the Linux kernel.

To print the kernel version of each node in your cluster, run:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kernelVersion}{"\n"}{end}'

Make sure all nodes have a kernel version >= 4.9.

Add the Grafana Helm repository

Use Helm to install Alloy. To add the Grafana Helm repository, run:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Verify that the repository was added successfully by running:

helm search repo grafana/alloy

The command returns a list of available versions of Alloy.

Create an Alloy configuration file

Create a file named values.yaml with the content from the sample configuration file.

alloy:
  configMap:
    create: true
    content: |
      discovery.kubernetes "local_pods" {
        selectors {
          field = "spec.nodeName=" + env("HOSTNAME")
          role = "pod"
        }
        role = "pod"
      }
      pyroscope.ebpf "instance" {
        forward_to = [pyroscope.write.endpoint.receiver]
        targets = discovery.kubernetes.local_pods.targets
      }
      pyroscope.write "endpoint" {
        endpoint {
          basic_auth {
            password = "<PASSWORD>"
            username = "<USERNAME>"
          }
          url = "<URL>"
        }
      }

  securityContext:
    privileged: true
    runAsGroup: 0
    runAsUser: 0

controller:
  hostPID: true

For information about configuring Alloy, refer to Grafana Alloy on Kubernetes.

For information about the specific blocks used, refer to the Grafana Alloy Reference.

Replace the <URL> placeholder with the appropriate server URL. This could be the Grafana Cloud URL or your own custom Pyroscope server URL.

If you need to send data to Grafana Cloud, you’ll have to configure HTTP Basic authentication. Replace <User> with your Grafana Cloud stack user and <Password> with your Grafana Cloud API key.

For more information, refer to the Configure the Grafana Pyroscope data source documentation.

Note
If you’re using your own Pyroscope server, you can remove the basic_auth section altogether.

Install Alloy

To install Alloy, run:

helm install pyroscope-ebpf grafana/alloy -f values.yaml

Once configured, Alloy starts collecting eBPF profiles and sends them to the Pyroscope server.

Verify profiles are received

To verify that the profiles are received by the Pyroscope server:

Go to the Pyroscope UI or Grafana Pyroscope data source.
Select a profile type and a service from the drop-down menu.

Considerations for profiling applications in containers

When profiling Python applications running in containers using the pyroscope.ebpf component in Alloy, consider the following:

Kernel version: Ensure that the host system’s kernel version is >= 4.9, as required by eBPF. This is crucial for the profiler to function correctly
Container privileges: The eBPF profiler requires certain privileges to access kernel features. Ensure that the container running the profiler has the necessary permissions. This typically involves setting the container to run in privileged mode or adjusting security contexts
Host PID namespace: The profiler may need access to the host’s PID namespace to correctly attach to processes. Ensure that the hostPID is set to true in your Kubernetes configuration
Network access: Ensure that the container has network access to send profiling data to the Pyroscope server. This may involve configuring network policies or service accounts

Set up eBPF profiling on Linux

Wed, 08 Apr 2026 14:38:28 +0000

Set up eBPF profiling on Linux

To set up eBPF profiling with Grafana Alloy on Linux, you need to:

Verify that your system meets the requirements.
Install Alloy.
Create an Alloy configuration file. For more information, refer to Configuration reference.
Run Alloy.
Finally, verify that profiles are received.

Before you begin

Before you begin, you need:

A Pyroscope server where Alloy can send profiling data.
Access to Grafana with the Grafana Pyroscope data source provisioned.

Note
If you don’t have a Grafana or a Pyroscope server, you can use the Grafana Cloud free plan to get started.

Verify system meets the requirements

The eBPF profiler requires a Linux kernel version >= 4.9 (due to BPF_PROG_TYPE_PERF_EVENT).

BPF_PROG_TYPE_PERF_EVENT is a type of eBPF program that can be attached to hardware or software events, such as performance monitoring counters or tracepoints, in the Linux kernel.

To print the kernel version of machine, run:

uname -r

Make sure you have a kernel version >= 4.9.

Install Alloy

Follow the installation instructions to download and install Alloy for your current Linux distribution.

Configure Alloy

To configure the Alloy eBPF profiler to profile local processes, use discovery.process component and add a default target in the pyroscope.ebpf component. All processes are profiled and grouped under the default target.

Create a file named alloy.config with the following content:

discovery.process "all" {

}

discovery.relabel "alloy" {
    targets = discovery.process.all.targets
    // Filter needed processes
    rule {
        source_labels = ["__meta_process_exe"]
        regex = ".*/alloy"
        action = "keep"
    }
}

pyroscope.ebpf "instance" {
 forward_to     = [pyroscope.write.endpoint.receiver]
 targets = discovery.relabel.alloy.output
}

pyroscope.scrape "local" {
  forward_to     = [pyroscope.write.endpoint.receiver]
  targets    = [
    {"__address__" = "localhost:12345", "service_name"="grafana/alloy"},
  ]
}

pyroscope.write "endpoint" {
 endpoint {
  basic_auth {
   password = "<PASSWORD>"
   username = "<USERNAME>"
  }
  url = "<URL>"
 }
 external_labels = {
  "env"      = "prod",
  "instance" = env("HOSTNAME"),
 }
}

For information about configuring Alloy, refer to Grafana Alloy on Kubernetes.

For information about the specific blocks used, refer to the Grafana Alloy Reference and discovery.process.

Replace the <URL> placeholder with the appropriate server URL. This could be the Grafana Cloud URL or your own custom Pyroscope server URL.

If you need to send data to Grafana Cloud, you’ll have to configure HTTP Basic authentication. Replace <User> with your Grafana Cloud stack user and <Password> with your Grafana Cloud API key.

For more information, refer to the Configure the Grafana Pyroscope data source documentation.

Note
If you’re using your own Pyroscope server, you can remove the basic_auth section altogether.

Start Alloy

Note
The eBPF profiler requires root privileges.

To start the Alloy, run:

alloy run alloy.config

If you see the following error:

level=error msg="component exited with error" component=pyroscope.ebpf.local_pods err="ebpf profiling session start: load bpf objects: field DisassociateCtty: program disassociate_ctty: map events: map create: operation not permitted (MEMLOCK may be too low, consider rlimit.RemoveMemlock)"

Make sure you’re running Alloy with root privileges which are required for the eBPF profiler to work.

Verify profiles are received

To verify that the profiles are received by the Pyroscope server, go to the Pyroscope UI or Grafana Pyroscope data source. Select a profile type and a service from the drop-down menu.

Setup eBPF Profiling on Docker

Wed, 08 Apr 2026 14:38:28 +0000

Setup eBPF Profiling on Docker

To set up eBPF profiling with Grafana Alloy on Linux, you need to:

Verify that your system meets the requirements.
Create an Alloy configuration file. For more information, refer to the Configuration reference.
Run Alloy.
Verify that profiles are received.

Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Before you begin

Before you begin, you need:

A Pyroscope server where Alloy can send profiling data.
Access to Grafana with the Grafana Pyroscope data source provisioned.
Docker Engine installed.

Note
If you don’t have a Grafana or a Pyroscope server, you can use the [Grafana Cloud][gcloud] free plan to get started.

Verify system requirements

The eBPF profiler requires a Linux kernel version >= 4.9 due to BPF_PROG_TYPE_PERF_EVENT.

BPF_PROG_TYPE_PERF_EVENT is a type of eBPF program that can be attached to hardware or software events, such as performance monitoring counters or tracepoints, in the Linux kernel.

To print the kernel version of your docker host, run:

docker info | grep Kernel

The kernel version must be 4.9 or later.

Configure Alloy

You can configure Alloy eBPF profiler to profile local containers. To do so, use the discovery.docker component to discover local containers and the pyroscope.ebpf component to profile them

For more information about the Alloy configuration, refer to the Alloy Components reference.

Create a file named alloy.config with the following content:

discovery.docker "local_containers" {
 host = "unix:///var/run/docker.sock"
}

pyroscope.ebpf "instance" {
 forward_to     = [pyroscope.write.endpoint.receiver]
 targets = discovery.docker.local_containers.targets
}

pyroscope.write "endpoint" {
 endpoint {
  basic_auth {
   password = "<PASSWORD>"
   username = "<USERNAME>"
  }
  url = "<URL>"
 }
 external_labels = {
  "env"      = "testing",
  "instance" = env("HOSTNAME"),
 }
}

Replace the <URL> placeholder with the appropriate server URL. This could be the Grafana Cloud URL or your own custom Pyroscope server URL.

If you need to send data to Grafana Cloud, you’ll have to configure HTTP Basic authentication. Replace <User> with your Grafana Cloud stack user and <Password> with your Grafana Cloud API key.

For more information, refer to the Configure the Grafana Pyroscope data source documentation.

Note
If you’re using your own Pyroscope server, you can remove the basic_auth section altogether.

Start Alloy

To start Alloy with Docker, run:

docker run \
  -v $PWD/alloy.config:/etc/alloy/alloy.config \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --pid=host \
  --privileged \
  -p 12345:12345 \
  grafana/alloy:latest \
    run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/alloy.config

Note
The --pid=host and --privileged flags are required to profile local containers with eBPF.

Verify profiles are received

To verify that the profiles are received by the Pyroscope server, go to the Pyroscope UI or Grafana Pyroscope data source. Then select a profile type and a service from the dropdown menu.

Configuration reference

Wed, 08 Apr 2026 14:38:28 +0000

Configuration reference

Grafana Alloy supports eBPF profiling. The configuration file is written in the River language and is composed of components that are used to collect, transform, and send data.

The pyroscope.ebpf component is used to collect application performance profiles via eBPF.

The pyroscope.ebpf runs on the host machine and collects stack traces associated with a process running on the current host.

Using the targets argument, you can specify which processes and containers to profile on the machine. The targets can be from discovery components such as discovery.process, discovery.kubernetes, discovery.docker, and discovery.dockerswarm.

You can use the discovery.relabel component to relabel discovered targets and set your own labels. For more information, refer to the Components documentation.

The forward_to parameter should point to a pyroscope.write component to send the collected profiles to your Pyroscope Server or Grafana Cloud.

Name	Type	Description	Default	Required
`targets`	`list(map(string))`	List of targets to group profiles by container id		yes
`forward_to`	`list(ProfilesReceiver)`	List of receivers to send collected profiles to.		yes
`collect_interval`	`duration`	How frequently to collect profiles	`15s`	no
`sample_rate`	`int`	How many times per second to collect profile samples	97	no
`pid_cache_size`	`int`	The size of the pid -> proc symbols table LRU cache	32	no
`build_id_cache_size`	`int`	The size of the elf file build id -> symbols table LRU cache	64	no
`same_file_cache_size`	`int`	The size of the elf file -> symbols table LRU cache	8	no
`container_id_cache_size`	`int`	The size of the pid -> container ID table LRU cache	1024	no
`collect_user_profile`	`bool`	A flag to enable/disable collection of userspace profiles	true	no
`collect_kernel_profile`	`bool`	A flag to enable/disable collection of kernelspace profiles	true	no
`demangle`	`string`	C++ demangle mode. Available options are: `none`, `simplified`, `templates`, `full`	`none`	no

Supported languages

The pyroscope.ebpf component supports the following languages:

Go with frame pointers enabled (default)
Rust with frame pointers enabled
C/C++ with frame pointers enabled
Python

Send data Grafana Cloud Profiles

When sending to Grafana Cloud Profiles, you can use the following pyroscope.write component configuration which makes uses of environment variables:

pyroscope.write "endpoint" {
    endpoint {
        basic_auth {
            password = env("GC_PASSWORD")
            username = env("GC_USER")
        }
        url = env("GC_URL")
    }
}

Ensure that you have appropriately configured the GC_URL, GC_USER, and GC_PASSWORD environment variables.

Profile collecting behavior

The pyroscope.ebpf component collects stack traces associated with a process running on the current host. You can use the sample_rate argument to define the number of stack traces collected per second. The default is 97.

The following labels are automatically injected into the collected profiles if you have not defined them. These labels can help you pin down a profiling target.

Label	Description
`service_name`	Pyroscope service name. It’s automatically selected from discovery meta labels, if possible. Otherwise, it defaults to `unspecified`.
`__name__`	pyroscope metric name. Defaults to `process_cpu`.
`__container_id__`	The container ID derived from target.

Privileges

You are required to run the agent as root and inside host pid namespace in order to pyroscope.ebpf component to work.

Targets

One of the following special labels must be included in each target of targets and the label must correspond to the container or process that is profiled:

__container_id__: The container ID.
__meta_docker_container_id: The ID of the Docker container.
__meta_kubernetes_pod_container_id: The ID of the Kubernetes pod container.
__process_pid__ : The process ID.

Each process is then associated with a specified target from the targets list, determined by a container ID or process PID.

If a process’s container ID matches a target’s container ID label, the stack traces are aggregated per target based on the container ID. If a process’s PID matches a target’s process PID label, the stack traces are aggregated per target based on the process PID. Otherwise, the process is not profiled.

Service name

The special label service_name is required and must always be present. If it’s not specified, it is attempted to be inferred from multiple sources:

__meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name pod annotation.
__meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
__meta_docker_container_name
__meta_dockerswarm_container_label_service_name and __meta_dockerswarm_service_name

If service_name is not specified and could not be inferred, it is set to unspecified.

Exposed Prometheus metrics

The pyroscope.ebpf component exposes the following Prometheus metrics:

pyroscope_fanout_latency (histogram): Write latency for sending to direct and indirect components.
pyroscope_ebpf_active_targets (gauge): Number of active targets the component tracks.
pyroscope_ebpf_profiling_sessions_total (counter): Number of profiling sessions completed.
pyroscope_ebpf_profiling_sessions_failing_total (counter): Number of profiling sessions failed.
pyroscope_ebpf_pprofs_total (counter): Number of pprof profiles collected by the ebpf component.

More information

Check out the following resources to learn more about eBPF profiling:

The pros and cons of eBPF profiling blog post (for more context on flame graphs below)
Demo showing breakdown of our examples cluster
Grafana Alloy
pyroscope.scrape
pyroscope.write
discovery.kubernetes
discovery.docker
discovery.relabel

Troubleshoot eBPF installation

Wed, 08 Apr 2026 14:38:28 +0000

Troubleshoot eBPF installation

Learn how to troubleshoot and resolve eBPF installation issues.

Profile interpreted languages

Profiling interpreted languages like Ruby, JavaScript, etc., isn’t ideal using this implementation. The JIT-compiled methods in these languages are typically not in ELF file format, demanding additional steps for profiling. For instance, using perf-map-agent and enabling frame pointers for Java.

Interpreted methods display the interpreter function’s name rather than the actual function.

Troubleshoot unknown symbols

Symbols are extracted from various sources, including:

The .symtab and .dynsym sections in the ELF file.
The .symtab and .dynsym sections in the debug ELF file.
The .gopclntab section in Go language ELF files.

The search for debug files follows gdb algorithm. For example, if the profiler wants to find the debug file for /lib/x86_64-linux-gnu/libc.so.6 with a .gnu_debuglink set to libc.so.6.debug and a build ID 0123456789abcdef. The following paths are examined:

/usr/lib/debug/.build-id/01/0123456789abcdef.debug
/lib/x86_64-linux-gnu/libc.so.6.debug
/lib/x86_64-linux-gnu/.debug/libc.so.6.debug
/usr/lib/debug/lib/x86_64-linux-gnu/libc.so.6.debug

Deal with unknown symbols

Unknown symbols in the profiles you’ve collected indicate that the profiler couldn’t access an ELF file associated with a given address in the trace.

This can occur for several reasons:

The process has terminated, making the ELF file inaccessible.
The ELF file is either corrupted or not recognized as an ELF file.
There is no corresponding ELF file entry in /proc/pid/maps for the address in the stack trace.

Address unresolved symbols

If you only see module names (e.g., /lib/x86_64-linux-gnu/libc.so.6) without corresponding function names, this indicates that the symbols couldn’t be mapped to their respective function names.

This can occur for several reasons:

The binary has been stripped, leaving no .symtab, .dynsym, or .gopclntab sections in the ELF file.
The debug file is missing or could not be located.

To fix this for your binaries, ensure that they are either not stripped or that you have separate debug files available. You can achieve this by running:

objcopy --only-keep-debug elf elf.debug
strip elf -o elf.stripped
objcopy --add-gnu-debuglink=elf.debug elf.stripped elf.debuglink

For system libraries, ensure that debug symbols are installed. On Ubuntu, for example, you can install debug symbols for libc by executing:

apt install libc6-dbg

Understand flat stack traces

If your profiles show many shallow stack traces, typically 1-2 frames deep, your binary might have been compiled without frame pointers.

To compile your code with frame pointers, include the -fno-omit-frame-pointer flag in your compiler options.

Ensure Python process data is discoverable

This error indicates that Pyroscope cannot locate required Python runtime symbols, potentially due to nonstandard library naming:

pyperf get python process data failed: missing symbols pyRuntimeAddr autoTLSkeyAddr

This can occur if the application build process uses custom naming for libraries, such as:

libpython3-custom.10.so.1.0

Pyroscope expects standard naming patterns like:

libpython3.10.so.1.0

To resolve this, ensure Python libraries follow standard naming conventions.