Ingest and analyze profile data on Grafana Labs

Understand profiling types and their uses in Pyroscope

Wed, 11 Mar 2026 09:33:08 +0100

Understand profiling types and their uses in Pyroscope

Profiling is an essential tool for understanding and optimizing application performance. In Pyroscope, various profiling types allow for an in-depth analysis of different aspects of your application. This guide will explore these types and explain their impact on your program.

Profiling types

In Pyroscope, profiling types refer to different dimensions of application performance analysis, focusing on specific aspects like CPU usage, memory allocation, or thread synchronization.

Available profile types

Various languages support different profiling types. Pyroscope supports the following profiling types as of v1.2.0:

Profile Type	Go	Java	.NET	Ruby	Python	Rust	Node.js	eBPF (Go)	eBPF (Python)
CPU	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Alloc Objects	Yes	Yes	Yes
Alloc Space	Yes	Yes	Yes
Inuse Objects	Yes
Inuse Space	Yes
Goroutines	Yes
Mutex Count	Yes		Yes
Mutex Duration	Yes		Yes
Block Count	Yes
Block Duration	Yes
Lock Count		Yes	Yes
Lock Duration		Yes	Yes
Exceptions			Yes
Wall			Yes
Heap							Yes

CPU profiling

CPU profiling measures the amount of CPU time consumed by different parts of your application code. High CPU usage can indicate inefficient code, leading to poor performance and increased operational costs. It’s used to identify and optimize CPU-intensive functions in your application.

When to use: To identify and optimize CPU-intensive functions
Flamegraph insight: The width of blocks indicates the CPU time consumed by each function

As you can see here the UI is showing a spike in CPU along with the flamegraph associated with that spike. Often times without profiling you may get similar insights from metrics, but with profiling you have more details into the specific cause of a spike in CPU usage at the line level

Memory allocation profiling

Memory allocation profiling tracks the amount and frequency of memory allocations by the application. Excessive or inefficient memory allocation can lead to memory leaks and high garbage collection overhead, impacting application efficiency.

Types: Alloc Objects, Alloc Space
When to use: For identifying and optimizing memory usage patterns
Flamegraph insight: Highlights functions where memory allocation is high

The timeline shows memory allocations over time and is great for debugging memory related issues. A common example is when a memory leak is created due to improper handling of memory in a function. This can be identified by looking at the timeline and seeing a gradual increase in memory allocations that never goes down. This is a clear indicator of a memory leak.

Without profiling this is may be something that is exhibited in metrics or OOM logs but with profiling you have more details into the specific function that is allocating the memory which is causiing the leak at the line level.

Goroutine profiling

Goroutines are lightweight threads in Go, used for concurrent operations. Goroutine profiling measures the usage and performance of these threads. Poor management can lead to issues like deadlocks and excessive resource usage.

When to use: Especially useful in Go applications for concurrency management
Flamegraph insight: Provides a view of goroutine distribution and issues

Mutex profiling

Mutex profiling involves analyzing mutex (mutual exclusion) locks, used to prevent simultaneous access to shared resources. Excessive or long-duration mutex locks can cause delays and reduced application throughput.

Types: Mutex Count, Mutex Duration
When to use: To optimize thread synchronization and reduce lock contention
Flamegraph insight: Shows frequency and duration of mutex operations

Block profiling

Block profiling measures the frequency and duration of blocking operations, where a thread is paused or delayed. Blocking can significantly slow down application processes, leading to performance bottlenecks.

Types: Block Count, Block Duration
When to use: To identify and reduce blocking delays
Flamegraph insight: Identifies where and how long threads are being blocked

Use the Pyroscope UI to explore profiling data

Wed, 11 Mar 2026 09:33:08 +0100

Use the Pyroscope UI to explore profiling data

Pyroscope’s UI is designed to make it easy to visualize and analyze profiling data. There are several different modes for viewing, analyzing, uploading, and comparing profiling data.

While code profiling has been a long-standing practice, continuous profiling represents a modern and more advanced approach to performance monitoring. This technique adds two critical dimensions to traditional profiles:

Time: Profiling data is collected continuously, providing a time-centric view that allows querying performance data from any point in the past.
Metadata: Profiles are enriched with metadata, adding contextual depth to the performance data.

These dimensions, coupled with the detailed nature of performance profiles, make continuous profiling a uniquely valuable tool. Pyroscope’s UI enhances this further by offering a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing.

In this UI reference, you’ll learn how Pyroscope parallels these other modern observability tools by providing a Prometheus-like querying experience. More importantly, you’ll learn how to use Pyroscope’s extensive UI features for a deeper insight into your application’s performance.

Key features of the Pyroscope UI

The following sections describe Pyroscope UI capabilities.

Tag Explorer

The Tag Explorer page is a vital part of Pyroscope’s UI, allowing users to navigate and analyze performance data through tags/labels. This feature is crucial for identifying performance anomalies and understanding the behavior of different application segments under various conditions. We intentionally don’t include a query language on this page as we built this page to be as intuitive as possible for users to use the UI to navigate and drill down into which tags are most interesting to them.

To use the Tag Explorer:

Select a tag to view the corresponding profiling data
Analyze the pie chart and the table of descriptive statistics to determine which tags if any are behaving abnormally
Select a tag to view the corresponding profiling data
Make use of the shortcuts to the single, comparison, and diff pages to further identify the root cause of the performance issue

Single view

The Single View page in Pyroscope’s UI is built for in-depth profile analysis. Here, you can explore a single flamegraph with multiple viewing options and functionalities:

Table view: Breaks down the profiling data into a sortable table format.
Sandwich view: Displays both the callers and callees for a selected function, offering a comprehensive view of function interactions.
Flamegraph view: Visualizes profiling data in a flamegraph format, allowing easy identification of resource-intensive functions.
Export & share: Options to export the flamegraph for offline analysis or share it via a flamegraph.com link for collaborative review.

The screenshot above shows a spike in CPU usage. Without profiling, we would go from a memory spike to digging through code or guessing what the cause of it is. However, with profiling, you can use the flamegraph and table to see exactly which function is most responsible for the spike. Often this will show up as a single node taking up a noticeably disproportionate width in the flamegraph as seen below with the “checkDriverAvailability” function.

However, in some instances it may be a function that is called many times and is taking up a large amount of space in the flamegraph. In this case, you can use the sandwich view to see that a logging function called throughout many functions in the codebase is the culprit.

Comparison page

The Comparison page facilitates side-by-side comparison of profiles either based on different label sets, different time periods, or both. This feature is extremely valuable for understanding the impact of changes or differences between do distinct queries of your application.

To run a comparison:

Select two different sets of labels (for exmaple, env:production vs. env:development) and or time periods, reflected by the sub-timelines above each flamegraph.
View the resulting flamegraphs side by side to identify disparities in performance.

There are many practical use cases for comparison for companies using Pyroscope. Some examples of labels below expressed as label:value are:

Feature flags: Compare application performance with feature_flag:a vs. feature_flag:b
Deployment environments: Contrast env:production vs. env:development
Release analysis: Examine commit:release-1 vs. commit:release-2
Region: Compare region:us-east-1 vs. region:us-west-1

Another example where time is more important than labels is when you want to compare two different time periods. For example, in investigating the cause of a memory leak you would see something like the following where the timeline shows an steadily increasing amount of memory allocations over time. This is a clear indicator of a memory leak.

You can then use the comparison page to compare the memory allocations between two different time periods where allocations were low and where allocations were high which would allow you to identify the function that is causing the memory leak.

Diff page: Identify changes with differential analysis

The Diff page is an extension of the comparison page, crucial for more easily visually showing the differences between two profiling data sets. It normalizes the data by comparing the percentage of total time spent in each function so that the resulting flamegraph is comparing the share of time spent in each function rather than the absolute amount of time spent in each function. This is important because it allows you to compare two different queries that may have different total amounts of time spent in each function.

Similar to a git diff it takes the flamegraphs from the comparison page and highlights the differences between the two flamegraphs where red represents an increase in CPU usage from the baseline to the comparison and green represents a decrease.

Using the same examples from above, here is a diff between two label sets:

Profile CLI

Wed, 11 Mar 2026 09:33:08 +0100

Profile CLI

profilecli is a command-line utility that enables various productivity flows such as:

Interacting with a running Pyroscope server to upload profiles, query data, and more
Inspecting Parquet files

Hint: Use the help command (profilecli help) to get a full list of capabilities as well as additional help information.

Install Profile CLI

You can install Profile CLI using a package or by compiling the code.

Install using a package

On macOS, you can install Profile CLI using HomeBrew:

brew install pyroscope-io/brew/profilecli

For other platforms, you can manually download the profilecli release asset for your operating system and architecture and make it executable.

For example, for Linux with the AMD64 architecture:

Download and extract the package (archive).

curl -fL https://github.com/grafana/pyroscope/releases/download/v1.1.5/profilecli_1.1.5_linux_amd64.tar.gz | tar xvz

Make profilecli executable:
Bash
```
chmod +x profilecli
```
Optional: Make profilecli reachable from anywhere:
Bash
```
sudo mv profilecli /usr/local/bin
```

Build from source code

To build from source code, you must have:

Go installed (> 1.19).
Either $GOPATH or $GOBIN configured and added to your PATH environment variable.

To build the source code:

Clone the repository.

git clone git@github.com:grafana/pyroscope.git

Run the Go install command to build and install the package.
Bash
```
cd pyroscope
go install ./cmd/profilecli
```
The command places the profilecli executable in $GOPATH/bin/ (or $GOBIN/) and make it available to use.

Common flags and environment variables

The profilecli commands that interact with a Pyroscope server require a server URL and optionally authentication details. These can be provided as command-line flags or environment variables.

Server URL

default: http://localhost:4040

The --url flag specifies the server against which the command will run. If using Grafana Cloud, an example URL could be https://profiles-prod-001.grafana.net. For local instances, the URL could look like http://localhost:4040.
Authentication details.

default: <empty>

If using Grafana Cloud or authentication is enabled on your Pyroscope server, you will need to provide a username and password using the --username and --password flags respectively. For Grafana Cloud, the username will be the Stack ID and the password the generated API token.

Environment variable naming

You can use environment variables to avoid passing flags to the command every time you use it, or to protect sensitive information. Environment variables have a PROFILECLI_ prefix. Here is an example of providing the server URL and credentials for the profilecli tool:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>
# now we can run a profilecli command without specifying the url or credentials:
profilecli <command>

Uploading a profile to a Pyroscope server using `profilecli`

Using profilecli streamlines the process of uploading profiles to Pyroscope, making it a convenient alternative to manual HTTP requests.

Prerequisites

Ensure you have profilecli installed on your system by following the installation steps above.
Have a profile file ready for upload. Note that you can only upload pprof files at this time.

Upload steps

Identify the pprof file.
- Path to your pprof file: path/to/your/pprof-file.pprof
Optional: Specify any extra labels.
- You can add additional labels to your uploaded profile using the --extra-labels flag.
- You can provide the name of the application that the profile was captured from via the service_name label (defaults to profilecli-upload). This will be useful when querying the data via profilecli or the UI.
- You can use the flag multiple times to add several labels.

Construct and execute the Upload command.

Here’s a basic command template:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>

profilecli upload --extra-labels=<label_name>=<label_value> <pprof_file_path>

Example command:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli upload path/to/your/pprof-file.pprof

Example command with extra labels:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli upload \
    --extra-labels=service_name=my_application_name \
    --extra-labels=cluster=us-east \
    path/to/your/pprof-file.pprof

Check for successful upload.
- After running the command, you should see a confirmation message indicating a successful upload. If there are any issues, profilecli provides error messages to help you troubleshoot.

Querying a Pyroscope server using `profilecli`

You can use the profilecli query command to look up the available profiles on a Pyroscope server and read actual profile data. This can be useful for debugging purposes or for integrating profiling in CI pipelines (for example to facilitate profile-guided optimization).

Looking up available profiles on a Pyroscope server

You can use the profilecli query series command to look up the available profiles on a Pyroscope server. By default, it queries the last hour of data, though this can be controlled with the --from and --to flags. You can narrow the results down with the --query flag. See profilecli help query series for more information.

Query series steps

Optional: Specify a Query and a Time Range.
- You can provide a label selector using the --query flag, for example: --query='{service_name="my_application_name"}'.
- You can provide a custom time range using the --from and --to flags, for example, --from="now-3h" --to="now".

Construct and execute the Query Series command.

Here’s a basic command template:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>

profilecli query series --query='{<label_name>="<label_value>"}'

Example command:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli query series --query='{service_name="my_application_name"}'

Example output:

{
    "__name__":"memory",
    "__period_type__":"space",
    "__period_unit__":"bytes",
    "__profile_type__":"memory:inuse_objects:count:space:bytes",
    "__service_name__":"my_application_name",
    "__type__":"inuse_objects",
    "__unit__":"count",
    "cluster":"eu-west-1",
    "service_name":"my_application_name"
 }

Reading a raw profile from a Pyroscope server

You can use the profilecli query merge command to retrieve a merged (aggregated) profile from a Pyroscope server. The command merges all samples found in the profile store for the specified query and time range. By default it looks for samples within the last hour, though this can be controlled with the --from and --to flags. The source data can be narrowed down with the --query flag in the same way as with the series command.

Query merge steps

Specify optional flags.
- You can provide a label selector using the --query flag, for example, --query='{service_name="my_application_name"}'.
- You can provide a custom time range using the --from and --to flags, for example, --from="now-3h" --to="now".
- You can specify the profile type via the --profile-type flag. The available profile types are listed in the output of the profilecli query series command.

Construct and execute the Query Merge command.

Here’s a basic command template:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>

profilecli query merge \
    --profile-type=<profile_type> \
    --query='{<label_name>="<label_value>"' \
    --from="<from>" --to="<to>"

Example command:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli query merge \
    --profile-type=memory:inuse_space:bytes:space:bytes \
    --query='{service_name="my_application_name"}' \
    --from="now-1h" --to="now"

Example output:

level=info msg="query aggregated profile from profile store" url=http://localhost:4040 from=2023-12-11T13:38:33.115683-04:00 to=2023-12-11T14:38:33.115684-04:00 query={} type=memory:inuse_space:bytes:space:bytes
PeriodType: space bytes
Period: 524288
Time: 2023-12-11 13:59:59.999 -0400 AST
Duration: 59m5
Samples:
inuse_space/bytes[dflt]
  115366240: 107 13 14 15 16 17 1 2 3
...

Ingest and analyze profile data on Grafana Labs

Understand profiling types and their uses in Pyroscope

Understand profiling types and their uses in Pyroscope

Profiling types

Available profile types

CPU profiling

Memory allocation profiling

Goroutine profiling

Mutex profiling

Block profiling

Use the Pyroscope UI to explore profiling data

Use the Pyroscope UI to explore profiling data

Key features of the Pyroscope UI

Tag Explorer

Single view

Comparison page

Diff page: Identify changes with differential analysis

Profile CLI

Profile CLI

Install Profile CLI

Install using a package

Build from source code

Common flags and environment variables

Environment variable naming

Uploading a profile to a Pyroscope server using profilecli

Prerequisites

Upload steps

Querying a Pyroscope server using profilecli

Looking up available profiles on a Pyroscope server

Query series steps

Reading a raw profile from a Pyroscope server

Query merge steps

Uploading a profile to a Pyroscope server using `profilecli`

Querying a Pyroscope server using `profilecli`