Documentation Grafana Alloy Reference Components pyroscope pyroscope.ebpf

pyroscope.ebpf

pyroscope.ebpf configures an eBPF profiling job for the current host. The component forwards the collected performance profiles to the receivers you list in forward_to.

The pyroscope.ebpf component embeds the grafana/opentelemetry-ebpf-profiler which is a fork of open-telemetry/opentelemetry-ebpf-profiler.

You can specify multiple pyroscope.ebpf components by giving them different labels, however it’s not recommended as it can lead to additional memory and CPU usage.

Required privileges

When running on Kubernetes without privileged: true, grant the following Linux capabilities and mount the kernel filesystem paths:

Mount /sys/kernel/tracing (on older kernel versions you might need /sys/kernel/debug instead) from the host as read-only volumes so the tracer can attach tracepoints.

Supported languages

• Native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host)
• Broad set of high-level languages (HotSpot JVM, Python, Ruby, PHP, Node.js, V8, Perl).

Usage

Alloy Copy

pyroscope.ebpf "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Arguments

The component configures and starts an eBPF profiling job to collect performance profiles from the current host.

You can use the following arguments with pyroscope.ebpf:

You must set the forward_to argument. Omitted fields take their default values.

Several arguments carry the label “Deprecated (no-op)”. These arguments once configured cache sizes and behaviors, but they no longer have any effect. Remove these arguments from your configuration.

Blocks

pyroscope.ebpf doesn’t support any blocks.

Exported fields

pyroscope.ebpf doesn’t export any fields.

Component health

pyroscope.ebpf reports itself as unhealthy if you give it an invalid configuration, or if the eBPF profiling session fails to start, for example because of missing privileges or an unsupported kernel.

Debug information

• elf_cache per build id and per same file symbol tables and their sizes in symbols count.
• pid_cache per process elf symbol tables and their sizes in symbols count.
• targets the active targets the component tracks.

Debug metrics

The following Prometheus metrics are exposed:

eBPF profiler internal metrics

The component also exposes internal metrics from the embedded eBPF profiler. The component emits these metrics only after recording each one at least once, so not all metrics appear on every host. The metrics carry an otel_scope_name="pyroscope.ebpf" label.

Notable metrics include:

Native unwinding

Interpreter unwinding

Symbolization

Process management

eBPF map state

The opentelemetry-ebpf-profiler metrics.json file defines the full list of metrics.

Profile collecting behavior

The pyroscope.ebpf component collects stack traces associated with a process running on the current host. You can use the sample_rate argument to define the number of stack traces collected per second. The default is 19.

The following labels are automatically injected into the collected profiles if you haven’t defined them. These labels can help you pin down a profiling target.

Alloy only sets otel.scope.name and otel.scope.version if they aren’t already present on the profile. otel.scope.version is only set when otel.scope.name matches the default value.

Targets

Each target in targets must include one of the following special labels, and the label must correspond to the container or process you profile:

• __container_id__: The container ID.
• __meta_docker_container_id: The ID of the Docker container.
• __meta_kubernetes_pod_container_id: The ID of the Kubernetes Pod container.
• __process_pid__ : The process ID.

Each process is then associated with a specified target from the targets list, determined by a container ID or process PID.

If a process’s container ID matches a target’s container ID label, the component aggregates the stack traces per target by container ID. If a process’s PID matches a target’s process PID label, the component aggregates the stack traces per target by process PID. Otherwise the process isn’t profiled.

Service name

Every profile carries a service_name label. You can set it on the target yourself. If you don’t, the component adds the label and infers a value, in order, from the following sources:

• __meta_kubernetes_pod_annotation_pyroscope_io_service_name, which is a pyroscope.io/service_name Pod annotation.
• __meta_kubernetes_namespace and __meta_kubernetes_pod_container_name, combined into ebpf/<namespace>/<container_name>.
• __meta_docker_container_name.
• __meta_dockerswarm_container_label_service_name.
• __meta_dockerswarm_service_name.

When you don’t set service_name and the component can’t infer a value, it uses unspecified.

Troubleshoot unknown symbols

The component extracts symbols from several sources, including:

• The .gopclntab section in Go language ELF files.
• The .symtab and .dynsym sections in the debug ELF file.
• The .symtab and .dynsym sections in the ELF file.

The search for debug files follows the GNU Debugger algorithm. For example, if the profiler wants to find the debug file for /lib/x86_64-linux-gnu/libc.so.6 with a .gnu_debuglink set to libc.so.6.debug and a build ID 0123456789abcdef. The component examines the following paths:

• /usr/lib/debug/.build-id/01/0123456789abcdef.debug
• /lib/x86_64-linux-gnu/libc.so.6.debug
• /lib/x86_64-linux-gnu/.debug/libc.so.6.debug
• /usr/lib/debug/lib/x86_64-linux-gnu/libc.so.6.debug

Deal with unknown symbols

Unknown symbols in the profiles you’ve collected indicate that the profiler couldn’t access an ELF file associated with a given address in the trace.

This can occur for several reasons:

• The process has terminated, making the ELF file inaccessible.
• The ELF file is either corrupted or not recognized as an ELF file.
• There is no corresponding ELF file entry in /proc/pid/maps for the address in the stack trace.

Address unresolved symbols

If you only see module names without corresponding function names, for example, /lib/x86_64-linux-gnu/libc.so.6, the component couldn’t map the symbols to their respective function names.

This can occur for several reasons:

• The build process stripped the binary, leaving no .symtab, .dynsym, or .gopclntab sections in the ELF file.
• The debug file is missing, or the component can’t locate it.

To fix this for your binaries, ensure that they’re either not stripped or that you have separate debug files available. You can achieve this by running:

Bash Copy

objcopy --only-keep-debug elf elf.debug
strip elf -o elf.stripped
objcopy --add-gnu-debuglink=elf.debug elf.stripped elf.debuglink

For system libraries, install the debug symbols. On Ubuntu, for example, you can install debug symbols for libc by executing:

Bash Copy

apt install libc6-dbg

Understand flat stack traces

If your profiles show many shallow stack traces, typically 1-2 frames deep, your compiler might have omitted frame pointers.

To compile your code with frame pointers, include the -fno-omit-frame-pointer flag in your compiler options.

Examples

The following examples show how to discover profiling targets and send the resulting profiles to a Pyroscope endpoint.

Kubernetes discovery

In the following example, the component collects performance profiles from Pods on the same node, discovered using discovery.kubernetes. Pod selection relies on the HOSTNAME environment variable, which is a Pod name if Alloy is used as an Alloy Helm chart. The example sets the service_name label to ebpf/{__meta_kubernetes_namespace}/{__meta_kubernetes_pod_container_name} from Kubernetes meta labels.

Alloy Copy

discovery.kubernetes "all_pods" {
  role = "pod"
  selectors {
    field = "spec.nodeName=" + sys.env("HOSTNAME")
    role = "pod"
  }
}

discovery.relabel "local_pods" {
  targets = discovery.kubernetes.all_pods.targets
  rule {
    action = "drop"
    regex = "Succeeded|Failed"
    source_labels = ["__meta_kubernetes_pod_phase"]
  }
  rule {
    action = "replace"
    regex = "(.*)@(.*)"
    replacement = "ebpf/${1}/${2}"
    separator = "@"
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    target_label = "service_name"
  }
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_namespace"]
    target_label = "namespace"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label = "pod"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label = "node"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label = "container"
  }
}
pyroscope.ebpf "local_pods" {
  forward_to = [ pyroscope.write.endpoint.receiver ]
  targets = discovery.relabel.local_pods.output
}

pyroscope.write "endpoint" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Docker discovery

The following example collects performance profiles from containers discovered by discovery.docker and ignores all other profiles collected from outside any docker container. The example sets the service_name label from the __meta_docker_container_name label.

Alloy Copy

discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "local_containers" {
  targets = discovery.docker.linux.targets
  rule {
    action = "replace"
    source_labels = ["__meta_docker_container_name"]
    target_label = "service_name"
  }
}

pyroscope.write "staging" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

pyroscope.ebpf "default" {
  forward_to   = [ pyroscope.write.staging.receiver ]
  targets      = discovery.relabel.local_containers.output
}

Collect profiles from probes

Use probe_links to collect stack traces whenever something calls the kernel or user-space probes you specify. Kernel probes use the format <PROBE_TYPE>:<SYMBOL>, and user-space probes use the format <PROBE_TYPE>:<EXECUTABLE_PATH>:<SYMBOL>. Supported probe types are kprobe, kretprobe, uprobe, and uretprobe.

The following example collects profiles when the kernel calls do_sys_open or when libc calls malloc:

Alloy Copy

pyroscope.ebpf "probes" {
	targets     = <TARGET_LIST>
	forward_to  = <RECEIVER_LIST>
	probe_links = [
		"kprobe:do_sys_open",
		"uprobe:/usr/lib/libc.so.6:malloc",
	]
}

Replace the following:

• <TARGET_LIST>: The list of process or container targets to profile.
• <RECEIVER_LIST>: The list of profile receivers to forward profiles to.

Correlate request profiles with Beyla traces and spans

When obi_process_context_enabled is true, pyroscope.ebpf uses a shared eBPF map under bpf_fs_root to correlate profile samples with traces generated by components compatible with OpenTelemetry eBPF Instrumentation (OBI), such as beyla.ebpf. Correlated samples include the trace_id and span_id labels. Components that share process context must use the same BPF filesystem root.

The following example uses beyla.ebpf to instrument requests to services listening on any port. It also profiles local processes and forwards the profiles to Pyroscope. Both components use their default BPF filesystem root at /sys/fs/bpf/.

Alloy Copy

beyla.ebpf "autoinstrument" {
	discovery {
		instrument {
			open_ports = "1-65535"
		}
	}

	output {
		traces = [otelcol.processor.batch.traces.input]
	}
}

otelcol.processor.batch "traces" {
	output {
		traces = [otelcol.exporter.otlphttp.traces.input]
	}
}

otelcol.exporter.otlphttp "traces" {
	client {
		endpoint = "<OTLP_TRACES_ENDPOINT>"
	}
}

discovery.process "all" { }

discovery.relabel "profiled_processes" {
	targets = discovery.process.all.targets

	rule {
		action        = "replace"
		source_labels = ["__meta_process_exe"]
		regex         = ".*/([^/]+)$"
		replacement   = "$1"
		target_label  = "service_name"
	}
}

pyroscope.ebpf "request_profiles" {
	targets                     = discovery.relabel.profiled_processes.output
	forward_to                  = [pyroscope.write.profiles.receiver]
	bpf_fs_root                 = "/sys/fs/bpf/"
	obi_process_context_enabled = true
}

pyroscope.write "profiles" {
	endpoint {
		url = "<PYROSCOPE_URL>"
	}
}

Replace the following:

• <OTLP_TRACES_ENDPOINT>: The OTLP HTTP endpoint to send traces to.
• <PYROSCOPE_URL>: The URL of the Pyroscope server to send profiles to.

Compatible components

pyroscope.ebpf can accept arguments from the following components:

• Components that export Targets
• Components that export Pyroscope ProfilesReceiver