This is documentation for the next version of Grafana Pyroscope documentation. For the latest stable release, go to the latest version.

Documentationbreadcrumb arrow Grafana Pyroscopebreadcrumb arrow Configure the client to send profilesbreadcrumb arrow OpenTelemetry eBPF profiler
Open source

OpenTelemetry eBPF profiler

Pyroscope supports receiving profiles from the OpenTelemetry eBPF profiler via OTLP. The eBPF profiler collects system-wide CPU profiles from all processes on a Linux host and sends them through the OpenTelemetry Collector pipeline to Pyroscope.

Before you begin

Consider the following limitations:

  • Protocol stability: The OpenTelemetry profiles signal is under active development. Breaking changes have occurred and may continue. Compatibility between the profiler, collector, and Pyroscope requires careful version management.
  • Symbolization: Function names may not resolve in flamegraphs for some programs. Symbol resolution is an area of active improvement.
  • Platform: The eBPF profiler requires Linux (amd64 or arm64) and privileged access to the host.

This feature is suitable for development and testing. Evaluate carefully before production use.

Architecture

The profiling pipeline has three components:

flowchart LR
    A["OTel eBPF Profiler
collects CPU profiles
system-wide via eBPF"] -- "OTLP gRPC" --> B["Pyroscope
stores & aggregates
profiling data"]
    B -- query --> C["Grafana
visualize as
flamegraphs"]
  1. OpenTelemetry eBPF Profiler runs as an OpenTelemetry Collector distribution with the profiling receiver. It attaches eBPF probes to collect CPU stack traces at 97 samples per second from every process on the host.
  2. Pyroscope receives profiles via OTLP gRPC on port 4040 and stores them.
  3. Grafana queries Pyroscope to visualize profiles as flamegraphs.

The profiler requires host PID namespace access and several host filesystem mounts (/proc, /sys/kernel, /lib/modules) to resolve stack traces.

Configure the collector

The profiler runs as a specialized OpenTelemetry Collector. Configure it with a profiling receiver and an otlp_grpc exporter pointing to Pyroscope:

YAML 
receivers:
  profiling:
    samples_per_second: 97

exporters:
  otlp_grpc:
    endpoint: pyroscope:4040
    tls:
      insecure: true

service:
  pipelines:
    profiles:
      receivers: [profiling]
      exporters: [otlp_grpc]

The collector must be started with the --feature-gates=service.profilesSupport flag.

Service name resolution

By default, the profiler sets process.executable.name on each profile but does not set service_name, which Pyroscope uses as the primary label. To map executable names to service names, configure Pyroscope with an ingestion relabeling rule:

YAML 
limits:
    ingestion_relabeling_rules:
        - action: labelmap
          regex: ^process.executable.name$
          replacement: service_name

Kubernetes metadata enrichment

In Kubernetes, you can add a k8sattributes processor to enrich profiles with pod, namespace, deployment, and node metadata:

YAML 
processors:
  k8sattributes/profiles:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.node.name
        - k8s.container.name
        - container.image.name
        - container.image.tag
        - service.name
        - service.namespace
        - service.instance.id
      otel_annotations: true
    pod_association:
      - sources:
          - from: resource_attribute
            name: container.id

service:
  pipelines:
    profiles:
      receivers: [profiling]
      processors: [k8sattributes/profiles]
      exporters: [otlp_grpc]

This requires a ServiceAccount with RBAC permissions to read pods, namespaces, nodes, and workload resources. See the example RBAC manifest.

Deploy with Docker Compose

A minimal Docker Compose setup runs the profiler, Pyroscope, and Grafana:

YAML 
services:
  otel-ebpf-profiler:
    image: otel/opentelemetry-collector-ebpf-profiler:0.147.0
    command:
      - --config=/etc/ebpf-profiler-config.yaml
      - --feature-gates=service.profilesSupport
    privileged: true
    pid: "host"
    volumes:
      - ./config/ebpf-profiler-config.yaml:/etc/ebpf-profiler-config.yaml:ro
      - /sys/kernel/debug:/sys/kernel/debug:ro
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
      - /proc:/proc:ro

  pyroscope:
    image: grafana/pyroscope:1.18.1
    command:
      - -self-profiling.disable-push=true
      - -config.file=/etc/pyroscope.yaml
    ports:
      - "4040:4040"

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_PLUGINS_PREINSTALL_SYNC=grafana-pyroscope-app
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    ports:
      - "3000:3000"

For a complete working example, see the Docker example.

Deploy on Kubernetes

On Kubernetes, deploy the profiler as a DaemonSet so it runs on every node:

YAML 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-ebpf-profiler
spec:
  selector:
    matchLabels:
      app: otel-ebpf-profiler
  template:
    metadata:
      labels:
        app: otel-ebpf-profiler
    spec:
      hostPID: true
      serviceAccountName: otel-ebpf-profiler
      containers:
        - name: profiler
          image: otel/opentelemetry-collector-ebpf-profiler:0.147.0
          args:
            - "--config=/etc/otel/config.yaml"
            - "--feature-gates=+service.profilesSupport"
          securityContext:
            privileged: true
          volumeMounts:
            - name: sys-kernel
              mountPath: /sys/kernel
              readOnly: true
            - name: proc
              mountPath: /proc
              readOnly: true
      volumes:
        - name: sys-kernel
          hostPath:
            path: /sys/kernel
        - name: proc
          hostPath:
            path: /proc
      tolerations:
        - operator: Exists

For a complete working example with kustomize, Pyroscope, Grafana, RBAC, and a sample workload, see the Kubernetes example.

Deploy it with:

Bash 
git clone --depth 1 --filter=tree:0 --no-checkout https://github.com/grafana/pyroscope.git
cd pyroscope
git sparse-checkout set examples/grafana-alloy-auto-instrumentation/ebpf-otel
git checkout
kubectl apply -k examples/grafana-alloy-auto-instrumentation/ebpf-otel/kubernetes/

Verify

After deploying, open Grafana (http://localhost:3000 for Docker, or port-forward in Kubernetes) and navigate to Explore with the Pyroscope data source. You should see profiles grouped by service_name appearing within a few minutes.

Profiles in Grafana