Pyroscope Symbolization

Symbolization is a critical process in profiling that transforms raw memory addresses into human-readable function names and source code paths. This process is necessary when working with compiled languages’ (C, C++, Rust, Go, etc.) binaries, which might not include debug information. Often this debug information is intentionally stripped to reduce binary size in production environments.

Unlike interpreted languages, compiled code typically requires this additional step to transform memory addresses into a meaningful context, allowing developers to accurately identify performance bottlenecks.

For example, without symbolization, a profile might show:

60.00%  [0x7f9b1c8e2a10]
25.00%  [0x7f9b1c8e3b20]
15.00%  [0x7f9b1c8e4c30]

After symbolization, the same profile might show:

60.00%  main.processData (/app/main.cpp:42)
25.00%  utils.parseInput (/app/utils.cpp:78)
15.00%  memory.allocateBuffer (/app/memory.cpp:156)

When is symbolization needed?

Symbolization is required in the following scenarios:

Native code profiling: Applications written in languages like C, C++, Rust, or Go with go that compile to native code.
System libraries: When profiling applications that use system libraries that don’t have symbolization information.
Stripped binaries: Production binaries that have been stripped of debug information to reduce size.
Mixed language applications: Applications that combine interpreted languages, like Python or Java, with native extensions.

Current symbolization support

In the current implementation, the symbolization feature has the following scope:

System libraries only: Symbolization is currently limited to system libraries and other non-customer code.
Customer code: Symbolization for customer code is not currently supported.
Function names and file paths: Current implementation provides function names, but line numbers and inline function information are not yet supported.
Debuginfod integration: For non-customer code, Pyroscope uses a configurable debuginfod server to retrieve the necessary debug information.
Automatic Detection: Pyroscope automatically detects when profiles need symbolization based on the presence of mappings without function information.
Per-Tenant Configuration: Symbolization can be enabled or disabled on a per-tenant basis.

How symbolization works in Pyroscope

Pyroscope automatically attempts to symbolize profiles during query execution when certain conditions are met. Here are the key points:

HasFunctions: false: Pyroscope only attempts symbolization on profile mappings that have HasFunctions: false. If a mapping already has function information, no symbolization is attempted.
Build ID availability: The binary’s build ID must be available on a public debuginfod server. Without this, Pyroscope cannot locate the necessary debug information.
Caching: After debug information is successfully retrieved, Pyroscope caches it to improve performance for subsequent queries involving the same binaries.

Process overview

When you query a profile, Pyroscope automatically:

Checks if symbolization is enabled for your account
Identifies mappings without function information (HasFunctions: false)
Attempts to retrieve debug information for those mappings from debuginfod servers
Updates the profile with symbolized function names if debug information is found
Caches results for faster future queries

Requirements for symbolization

Debug information

For symbolization to work, debug information must be available for the binaries being profiled:

Build IDs: Binaries must have valid build IDs that can be used to locate debug information.
Debug packages: Debug information packages should be available through debuginfod servers.
debuginfod server: A configured debuginfod server that can provide debug information.

The `debuginfod` server

A debuginfod server is a service that provides debug information for binaries based on their build ID. Options include:

Public servers:
- Default: https://debuginfod.elfutils.org/ (used if no URL specified)
- Fedora: https://debuginfod.fedoraproject.org
- Ubuntu: https://debuginfod.ubuntu.com
Private servers: Organizations can set up their own debuginfod servers. Note that the current implementation does not support authentication, so private servers must be accessible without credentials or use network-level access control.

Troubleshoot symbolization issues

When symbolization isn’t working

Follow these debugging steps If you expect symbolization to work but still see memory addresses or “unknown” in Profiles Drilldown.

Step 1: Download and inspect the profile

Download the pprof file: Export the profile data as a pprof file from the Pyroscope UI.
Inspect profile mappings: Use go tool pprof -raw to examine the mapping information:
Bash
```
go tool pprof -raw your-profile.pprof | grep -A 10 "^Mapping"
```
This command shows all binary mappings in the profile with their build IDs and HasFunctions status.

Step 2: Verify build IDs and debug information availability

Identify mappings with build IDs: In the output from the previous command, look for:
- Build IDs: Mappings that can be symbolized show a build ID (long hexadecimal string)
- Missing build IDs: Some mappings may not have build IDs at all
Example output:
```
1: 0x403000/0x9829000/0x3000 alloy abc123ea7079ec2bb48e555a4b03607fa99def8
2: 0x0/0x2000/0x0 linux-vdso.1.so
3: 0x400000/0x12507000/0x0 alloy xyz456069a7ac390778b9eadf0e100440855f88c
```
In this example:
- Mapping 1 and 3 have build IDs (abc123ea... and xyz456069...) and can potentially be symbolized
- Mapping 2 (linux-vdso.1.so) has no build ID and cannot be symbolized
Test debug information availability: For each build ID, check if debug info is available:
Quick check (recommended):
Bash
```
curl -I "https://debuginfod.elfutils.org/buildid/YOUR_BUILD_ID/debuginfo"
```
Download debug info (if you want to verify it’s complete):
Bash
```
curl -v -L -o symbols.debug "https://debuginfod.elfutils.org/buildid/YOUR_BUILD_ID/debuginfo"
```
- 200 OK: Debug information is available for symbolization
- 404 Not Found: Debug info isn’t available, symbolization won’t work for this mapping

Step 3: Verify Symbolization Requirements

Ensure your setup meets these requirements:

HasFunctions: false: Only mappings with HasFunctions: false are symbolized
Valid build IDs: Each binary mapping must have a build ID
Public debug info: Build IDs must be available on public debuginfod servers
System libraries only: Custom application code symbolization is not currently supported

Advanced troubleshooting: Direct `HasFunctions` inspection

If you need to see the exact HasFunctions status for each mapping (the field Pyroscope uses to decide whether to attempt symbolization), you can use this Go program:

package main

import (
    "fmt"
    "log"
    "os"
    
    "github.com/google/pprof/profile"
)

func main() {
    f, err := os.Open("your_profile.pprof")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()
    
    p, err := profile.Parse(f)
    if err != nil {
        log.Fatal(err)
    }
    
    for _, mapping := range p.Mapping {
        fmt.Printf("ID: %d, File: %s, HasFunctions: %t, BuildID: %s\n", 
            mapping.ID, mapping.File, mapping.HasFunctions, mapping.BuildID)
    }
}

This shows exactly what Pyroscope sees. Pyroscope only attempts to symbolize mappings where HasFunctions: false.

Common issues and solutions

Issue	Symptoms	Solution
No build ID	Addresses shown instead of symbols	Ensure binaries are built with build IDs
Debug info not found	Symbolization attempted but fails	Verify debug packages are available on `debuginfod` server
HasFunctions: true	No symbolization attempted	Profile mappings should have `HasFunctions: false`
Network connectivity	Symbolization timeouts	Check access to `debuginfod` servers