Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Grafana Pyroscope Configure the client Grafana Agent Profiling with eBPF Configuration Reference

Configuration

Grafana Agent supports eBPF profiling in Flow mode. The configuration file is written in the River language and is composed of components that are used to collect, transform, and send data.

The pyroscope.ebpf component is used to collect application performance profiles via eBPF.

The pyroscope.ebpf runs on the host machine and collects stack traces associated with a process running on the current host.

Using the targets argument, you can specify which processes and containers to profile on the machine. The targets can be from discovery components such as dicovery.kubernetes, discovery.docker and discovery.dockerswarm. To relabel discovered targets and set your own labels you can use the discovery.relabel component. For more information, see Components.

The forward_to parameter should point to a pyroscope.write component to send the collected profiles to your Pyroscope Server or Grafana Cloud.

Supported languages

The pyroscope.ebpf component supports the following languages:

• Go
• Rust
• C/C++ with frame pointers enabled
• Python

Send data Grafana Cloud Profiles

When sending to Grafana Cloud Profiles, you can use the following pyroscope.write component configuration which makes uses of environment variables:

river Copy

pyroscope.write "endpoint" {
    endpoint {
        basic_auth {
            password = env("GC_PASSWORD")
            username = env("GC_USER")
        }
        url = env("GC_URL")
    }
}

Ensure that you have appropriately configured the GC_URL, GC_USER, and GC_PASSWORD environment variables.

Profile collecting behavior

The pyroscope.ebpf component collects stack traces associated with a process running on the current host. You can use the sample_rate argument to define the number of stack traces collected per second. The default is 97.

The following labels are automatically injected into the collected profiles if you have not defined them. These labels can help you pin down a profiling target.

Privileges

You are required to run the agent as root and inside host pid namespace in order to pyroscope.ebpf component to work.

Container ID

Each collected stack trace is then associated with a specified target from the targets list, determined by a container ID. This association process involves checking the __container_id__, __meta_docker_container_id, __meta_dockerswarm_task_container_id and __meta_kubernetes_pod_container_id labels of a target against the /proc/{pid}/cgroup of a process.

If a corresponding container ID is found, the stack traces are aggregated per target based on the container ID. If a container ID is not found, the stack trace is associated with a default_target.

Any stack traces not associated with a listed target are ignored.

Service name

The special label service_name is required and must always be present. If it’s not specified, it is attempted to be inferred from multiple sources:

• __meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name pod annotation.
• __meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
• __meta_docker_container_name
• __meta_dockerswarm_container_label_service_name and __meta_dockerswarm_service_name

If service_name is not specified and could not be inferred, it is set to unspecified.

Exposed Prometheus metrics

The pyroscope.ebpf component exposes the following Prometheus metrics:

• pyroscope_fanout_latency (histogram): Write latency for sending to direct and indirect components.
• pyroscope_ebpf_active_targets (gauge): Number of active targets the component tracks.
• pyroscope_ebpf_profiling_sessions_total (counter): Number of profiling sessions completed.
• pyroscope_ebpf_profiling_sessions_failing_total (counter): Number of profiling sessions failed.
• pyroscope_ebpf_pprofs_total (counter): Number of pprof profiles collected by the ebpf component.

More information

Check out the following resources to learn more about eBPF profiling:

• The pros and cons of eBPF profiling blog post (for more context on flamegraphs below)
• Demo showing breakdown of our examples cluster
• Grafana Agent documentation for pyroscope.ebpf, pyroscope.write, discovery.kubernetes, discovery.relabel components