Documentationbreadcrumb arrow Grafana Pyroscopebreadcrumb arrow Configure the client to send profilesbreadcrumb arrow Grafana Alloybreadcrumb arrow Set up profiling with eBPFbreadcrumb arrow Troubleshoot
Open source

Troubleshoot eBPF installation

Learn how to troubleshoot and resolve eBPF installation issues.

Profile interpreted languages

Profiling interpreted languages like Ruby, JavaScript, etc., isn’t ideal using this implementation. The JIT-compiled methods in these languages are typically not in ELF file format, demanding additional steps for profiling. For instance, using perf-map-agent and enabling frame pointers for Java.

Interpreted methods display the interpreter function’s name rather than the actual function.

Troubleshoot unknown symbols

Symbols are extracted from various sources, including:

  • The .symtab and .dynsym sections in the ELF file.
  • The .symtab and .dynsym sections in the debug ELF file.
  • The .gopclntab section in Go language ELF files.

The search for debug files follows gdb algorithm. For example, if the profiler wants to find the debug file for /lib/x86_64-linux-gnu/libc.so.6 with a .gnu_debuglink set to libc.so.6.debug and a build ID 0123456789abcdef. The following paths are examined:

  • /usr/lib/debug/.build-id/01/0123456789abcdef.debug
  • /lib/x86_64-linux-gnu/libc.so.6.debug
  • /lib/x86_64-linux-gnu/.debug/libc.so.6.debug
  • /usr/lib/debug/lib/x86_64-linux-gnu/libc.so.6.debug

Deal with unknown symbols

Unknown symbols in the profiles you’ve collected indicate that the profiler couldn’t access an ELF file associated with a given address in the trace.

This can occur for several reasons:

  • The process has terminated, making the ELF file inaccessible.
  • The ELF file is either corrupted or not recognized as an ELF file.
  • There is no corresponding ELF file entry in /proc/pid/maps for the address in the stack trace.

Address unresolved symbols

If you only see module names (e.g., /lib/x86_64-linux-gnu/libc.so.6) without corresponding function names, this indicates that the symbols couldn’t be mapped to their respective function names.

This can occur for several reasons:

  • The binary has been stripped, leaving no .symtab, .dynsym, or .gopclntab sections in the ELF file.
  • The debug file is missing or could not be located.

To fix this for your binaries, ensure that they are either not stripped or that you have separate debug files available. You can achieve this by running:

Bash 
objcopy --only-keep-debug elf elf.debug
strip elf -o elf.stripped
objcopy --add-gnu-debuglink=elf.debug elf.stripped elf.debuglink

For system libraries, ensure that debug symbols are installed. On Ubuntu, for example, you can install debug symbols for libc by executing:

Bash 
apt install libc6-dbg

Understand flat stack traces

If your profiles show many shallow stack traces, typically 1-2 frames deep, your binary might have been compiled without frame pointers.

To compile your code with frame pointers, include the -fno-omit-frame-pointer flag in your compiler options.

Ensure Python process data is discoverable

This error indicates that Pyroscope cannot locate required Python runtime symbols, potentially due to nonstandard library naming:

pyperf get python process data failed: missing symbols pyRuntimeAddr autoTLSkeyAddr

This can occur if the application build process uses custom naming for libraries, such as:

  • libpython3-custom.10.so.1.0

Pyroscope expects standard naming patterns like:

  • libpython3.10.so.1.0

To resolve this, ensure Python libraries follow standard naming conventions.