Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Open source RSS

Configuration

Grafana Agent supports eBPF profiling in Flow mode. The configuration file is written in the River language and is composed of components that are used to collect, transform, and send data.

The pyroscope.ebpf component is used to collect application performance profiles via eBPF.

Pyroscope ebpf diagram

The pyroscope.ebpf runs on the host machine and collects stack traces associated with a process running on the current host.

Using the targets argument, you can specify which processes and containers to profile on the machine. The targets can be from discovery components such as dicovery.kubernetes, discovery.docker and discovery.dockerswarm. To relabel discovered targets and set your own labels you can use the discovery.relabel component. For more information, see Components.

The forward_to parameter should point to a pyroscope.write component to send the collected profiles to your Pyroscope Server or Grafana Cloud.

NameTypeDescriptionDefaultRequired
targetslist(map(string))List of targets to group profiles by container idyes
forward_tolist(ProfilesReceiver)List of receivers to send collected profiles to.yes
collect_intervaldurationHow frequently to collect profiles15sno
sample_rateintHow many times per second to collect profile samples97no
pid_cache_sizeintThe size of the pid -> proc symbols table LRU cache32no
build_id_cache_sizeintThe size of the elf file build id -> symbols table LRU cache64no
same_file_cache_sizeintThe size of the elf file -> symbols table LRU cache8no
container_id_cache_sizeintThe size of the pid -> container ID table LRU cache1024no
collect_user_profileboolA flag to enable/disable collection of userspace profilestrueno
collect_kernel_profileboolA flag to enable/disable collection of kernelspace profilestrueno
demanglestringC++ demangle mode. Available options are: none, simplified, templates, fullnoneno

Supported languages

The pyroscope.ebpf component supports the following languages:

  • Go
  • Rust
  • C/C++ with frame pointers enabled
  • Python

Send data Grafana Cloud Profiles

When sending to Grafana Cloud Profiles, you can use the following pyroscope.write component configuration which makes uses of environment variables:

river
pyroscope.write "endpoint" {
    endpoint {
        basic_auth {
            password = env("GC_PASSWORD")
            username = env("GC_USER")
        }
        url = env("GC_URL")
    }
}

Ensure that you have appropriately configured the GC_URL, GC_USER, and GC_PASSWORD environment variables.

Profile collecting behavior

The pyroscope.ebpf component collects stack traces associated with a process running on the current host. You can use the sample_rate argument to define the number of stack traces collected per second. The default is 97.

The following labels are automatically injected into the collected profiles if you have not defined them. These labels can help you pin down a profiling target.

LabelDescription
service_namePyroscope service name. It’s automatically selected from discovery meta labels, if possible. Otherwise, it defaults to unspecified.
__name__pyroscope metric name. Defaults to process_cpu.
__container_id__The container ID derived from target.

Privileges

You are required to run the agent as root and inside host pid namespace in order to pyroscope.ebpf component to work.

Container ID

Each collected stack trace is then associated with a specified target from the targets list, determined by a container ID. This association process involves checking the __container_id__, __meta_docker_container_id, __meta_dockerswarm_task_container_id and __meta_kubernetes_pod_container_id labels of a target against the /proc/{pid}/cgroup of a process.

If a corresponding container ID is found, the stack traces are aggregated per target based on the container ID. If a container ID is not found, the stack trace is associated with a default_target.

Any stack traces not associated with a listed target are ignored.

Service name

The special label service_name is required and must always be present. If it’s not specified, it is attempted to be inferred from multiple sources:

  • __meta_kubernetes_pod_annotation_pyroscope_io_service_name which is a pyroscope.io/service_name pod annotation.
  • __meta_kubernetes_namespace and __meta_kubernetes_pod_container_name
  • __meta_docker_container_name
  • __meta_dockerswarm_container_label_service_name and __meta_dockerswarm_service_name

If service_name is not specified and could not be inferred, it is set to unspecified.

Exposed Prometheus metrics

The pyroscope.ebpf component exposes the following Prometheus metrics:

  • pyroscope_fanout_latency (histogram): Write latency for sending to direct and indirect components.
  • pyroscope_ebpf_active_targets (gauge): Number of active targets the component tracks.
  • pyroscope_ebpf_profiling_sessions_total (counter): Number of profiling sessions completed.
  • pyroscope_ebpf_profiling_sessions_failing_total (counter): Number of profiling sessions failed.
  • pyroscope_ebpf_pprofs_total (counter): Number of pprof profiles collected by the ebpf component.

More information

Check out the following resources to learn more about eBPF profiling: