View and analyze profile data on Grafana Labs

Understand profiling types and their uses in Pyroscope

Thu, 29 Feb 2024 16:10:16 +0000

Understand profiling types and their uses in Pyroscope

Profiling is an essential tool for understanding and optimizing application performance. In Pyroscope, various profiling types allow for an in-depth analysis of different aspects of your application. This guide will explore these types and explain their impact on your program.

Profiling types

In Pyroscope, profiling types refer to different dimensions of application performance analysis, focusing on specific aspects like CPU usage, memory allocation, or thread synchronization.

Available profile types

Various languages support different profiling types. Pyroscope supports the following profiling types as of v1.2.0:

Profile Type	Go	Java	.NET	Ruby	Python	Rust	Node.js	eBPF (Go)	eBPF (Python)
CPU	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Alloc Objects	Yes	Yes	Yes
Alloc Space	Yes	Yes	Yes
Inuse Objects	Yes
Inuse Space	Yes
Goroutines	Yes
Mutex Count	Yes		Yes
Mutex Duration	Yes		Yes
Block Count	Yes
Block Duration	Yes
Lock Count		Yes	Yes
Lock Duration		Yes	Yes
Exceptions			Yes
Wall			Yes
Heap							Yes

CPU profiling

CPU profiling measures the amount of CPU time consumed by different parts of your application code. High CPU usage can indicate inefficient code, leading to poor performance and increased operational costs. It’s used to identify and optimize CPU-intensive functions in your application.

When to use: To identify and optimize CPU-intensive functions
Flamegraph insight: The width of blocks indicates the CPU time consumed by each function

As you can see here the UI is showing a spike in CPU along with the flamegraph associated with that spike. Often times without profiling you may get similar insights from metrics, but with profiling you have more details into the specific cause of a spike in CPU usage at the line level

Memory allocation profiling

Memory allocation profiling tracks the amount and frequency of memory allocations by the application. Excessive or inefficient memory allocation can lead to memory leaks and high garbage collection overhead, impacting application efficiency.

Types: Alloc Objects, Alloc Space
When to use: For identifying and optimizing memory usage patterns
Flamegraph insight: Highlights functions where memory allocation is high

The timeline shows memory allocations over time and is great for debugging memory related issues. A common example is when a memory leak is created due to improper handling of memory in a function. This can be identified by looking at the timeline and seeing a gradual increase in memory allocations that never goes down. This is a clear indicator of a memory leak.

Without profiling this is may be something that is exhibited in metrics or OOM logs but with profiling you have more details into the specific function that is allocating the memory which is causiing the leak at the line level.

Goroutine profiling

Goroutines are lightweight threads in Go, used for concurrent operations. Goroutine profiling measures the usage and performance of these threads. Poor management can lead to issues like deadlocks and excessive resource usage.

When to use: Especially useful in Go applications for concurrency management
Flamegraph insight: Provides a view of goroutine distribution and issues

Mutex profiling

Mutex profiling involves analyzing mutex (mutual exclusion) locks, used to prevent simultaneous access to shared resources. Excessive or long-duration mutex locks can cause delays and reduced application throughput.

Types: Mutex Count, Mutex Duration
When to use: To optimize thread synchronization and reduce lock contention
Flamegraph insight: Shows frequency and duration of mutex operations

Block profiling

Block profiling measures the frequency and duration of blocking operations, where a thread is paused or delayed. Blocking can significantly slow down application processes, leading to performance bottlenecks.

Types: Block Count, Block Duration
When to use: To identify and reduce blocking delays
Flamegraph insight: Identifies where and how long threads are being blocked

Flamegraphs: Visualizing performance data

Thu, 29 Feb 2024 16:10:16 +0000

Flamegraphs: Visualizing performance data

A fundamental aspect of continuous profiling is the flamegraph, a convenient way to visualize performance data. These graphs provide a clear, intuitive understanding of resource allocation and bottlenecks within the application. Pyroscope extends this functionality with additional visualization formats like tree graphs and top lists.

How is a flamegraph created?

This diagram shows how code is turned into a flamegraph. In this case Pyroscope would sample the stacktrace of your application to understand how many CPU cycles are being spent in each function. It would then aggregate this data and turn it into a flamegraph. This is a very simplified example but it gives you an idea of how Pyroscope works.

What does a flamegraph represent?

Horizontally, the flamegraph represents 100% of the time that this application was running. The width of each node represents the amount of time spent in that function. The wider the node, the more time spent in that function. The narrower the node, the less time spent in that function.

Vertically, the nodes in the flamegraph represent the heirarchy of which functions were called and how much time was spent in each function. The top node is the root node and represents the total amount of time spent in the application. The nodes below it represent the functions that were called and how much time was spent in each function. The nodes below those represent the functions that were called from those functions and how much time was spent in each function. This continues until you reach the bottom of the flamegraph.

This is a CPU profile, but profiles can represent many other types of resource such as memory, network, disk, etc.

Use the Pyroscope UI to explore profiling data

Thu, 29 Feb 2024 16:10:16 +0000

Use the Pyroscope UI to explore profiling data

Pyroscope’s UI is designed to make it easy to visualize and analyze profiling data. There are several different modes for viewing, analyzing, uploading, and comparing profiling data.

While code profiling has been a long-standing practice, continuous profiling represents a modern and more advanced approach to performance monitoring. This technique adds two critical dimensions to traditional profiles:

Time: Profiling data is collected continuously, providing a time-centric view that allows querying performance data from any point in the past.
Metadata: Profiles are enriched with metadata, adding contextual depth to the performance data.

These dimensions, coupled with the detailed nature of performance profiles, make continuous profiling a uniquely valuable tool. Pyroscope’s UI enhances this further by offering a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing.

In this UI reference, you’ll learn how Pyroscope parallels these other modern observability tools by providing a Prometheus-like querying experience. More importantly, you’ll learn how to use Pyroscope’s extensive UI features for a deeper insight into your application’s performance.

Key features of the Pyroscope UI

The following sections describe Pyroscope UI capabilities.

Tag Explorer

The Tag Explorer page is a vital part of Pyroscope’s UI, allowing users to navigate and analyze performance data through tags/labels. This feature is crucial for identifying performance anomalies and understanding the behavior of different application segments under various conditions. We intentionally don’t include a query language on this page as we built this page to be as intuitive as possible for users to use the UI to navigate and drill down into which tags are most interesting to them.

To use the Tag Explorer:

Select a tag to view the corresponding profiling data
Analyze the pie chart and the table of descriptive statistics to determine which tags if any are behaving abnormally
Select a tag to view the corresponding profiling data
Make use of the shortcuts to the single, comparison, and diff pages to further identify the root cause of the performance issue

Single view

The Single View page in Pyroscope’s UI is built for in-depth profile analysis. Here, you can explore a single flamegraph with multiple viewing options and functionalities:

Table view: Breaks down the profiling data into a sortable table format.
Sandwich view: Displays both the callers and callees for a selected function, offering a comprehensive view of function interactions.
Flamegraph view: Visualizes profiling data in a flamegraph format, allowing easy identification of resource-intensive functions.
Export & share: Options to export the flamegraph for offline analysis or share it via a flamegraph.com link for collaborative review.

The screenshot above shows a spike in CPU usage. Without profiling, we would go from a memory spike to digging through code or guessing what the cause of it is. However, with profiling, you can use the flamegraph and table to see exactly which function is most responsible for the spike. Often this will show up as a single node taking up a noticeably disproportionate width in the flamegraph as seen below with the “checkDriverAvailability” function.

However, in some instances it may be a function that is called many times and is taking up a large amount of space in the flamegraph. In this case, you can use the sandwich view to see that a logging function called throughout many functions in the codebase is the culprit.

Comparison page

The Comparison page facilitates side-by-side comparison of profiles either based on different label sets, different time periods, or both. This feature is extremely valuable for understanding the impact of changes or differences between do distinct queries of your application.

To run a comparison:

Select two different sets of labels (for exmaple, env:production vs. env:development) and or time periods, reflected by the sub-timelines above each flamegraph.
View the resulting flamegraphs side by side to identify disparities in performance.

There are many practical use cases for comparison for companies using Pyroscope. Some examples of labels below expressed as label:value are:

Feature flags: Compare application performance with feature_flag:a vs. feature_flag:b
Deployment environments: Contrast env:production vs. env:development
Release analysis: Examine commit:release-1 vs. commit:release-2
Region: Compare region:us-east-1 vs. region:us-west-1

Another example where time is more important than labels is when you want to compare two different time periods. For example, in investigating the cause of a memory leak you would see something like the following where the timeline shows an steadily increasing amount of memory allocations over time. This is a clear indicator of a memory leak.

You can then use the comparison page to compare the memory allocations between two different time periods where allocations were low and where allocations were high which would allow you to identify the function that is causing the memory leak.

Diff page: Identify changes with differential analysis

The Diff page is an extension of the comparison page, crucial for more easily visually showing the differences between two profiling data sets. It normalizes the data by comparing the percentage of total time spent in each function so that the resulting flamegraph is comparing the share of time spent in each function rather than the absolute amount of time spent in each function. This is important because it allows you to compare two different queries that may have different total amounts of time spent in each function.

Similar to a git diff it takes the flamegraphs from the comparison page and highlights the differences between the two flamegraphs where red represents an increase in CPU usage from the baseline to the comparison and green represents a decrease.

Using the same examples from above, here is a diff between two label sets:

Understand 'self' vs. 'total' metrics in profiling with Pyroscope

Thu, 29 Feb 2024 16:10:16 +0000

Understand ‘self’ vs. ’total’ metrics in profiling with Pyroscope

Profiling in Pyroscope provideds many different ways of analyzing your profiling data. One of the key pieces of this analysis are the metrics ‘self’ and ’total’, whose understanding is key for accurate interpretation of profiling data in both CPU and memory contexts. These metrics can be seen both within the table and the flamegraph view of the UI.

Self

‘Self’ refers to the resource usage (CPU time, memory allocation, etc.) directly attributed to a specific function or a code segment, excluding the resources used by its sub-functions or calls

This metric helps isolate the direct impact of a specific code block, making it crucial for pinpointing primary resource consumers

Total

‘Total’ encompasses the combined resource usage of a function along with all the functions it calls

It provides a holistic view of a function’s overall resource consumption, essential for understanding cumulative impacts

‘Self’ and ‘Total’ in CPU profiling

In CPU profiling, ‘self’ indicates the CPU time consumed directly by the function, crucial for identifying functions with high CPU demand.

The ’total’ CPU time includes time spent in the function itself plus time in all called functions, highlighting comprehensive CPU usage

The example below demonstrates a simplified pseudocode representation of a CPU-intensive process. This illustrates how ‘self’ and ’total’ time would be calculated for different functions in a typical application. The following diagram provides a visual representation of these concepts.

def handle_request():
    # Root function representing the total request handling process
    parse_json()
    process_data()

def parse_json():
    # Function for parsing JSON data
    validate()

def validate():
    # Function for validating the parsed JSON schema
    # Schema validation logic

def process_data():
    # Function for processing data
    apply()

def apply():
    # Function for applying transformations to data
    # Transformation logic

# Simulate a request handling
handle_request()

‘Self’ and ‘Total’ in memory profiling

Self in Memory: For memory profiling, ‘self’ measures the memory allocated by the function itself, vital for detecting direct memory allocation issues.
Total in Memory: ‘Total’ memory includes allocations by the function and its called functions, essential for assessing overall memory footprint.

The same example from the CPU profiling section can be used to illustrate the concepts of ‘self’ and ’total’ in memory profiling, just with memory units instead of CPU.

Conclusion

Grasping the distinction between ‘self’ and ’total’ metrics is fundamental for effective performance analysis in Pyroscope. Whether in CPU or memory profiling (or any other type), these metrics provide value insights for optimizing applications and enhancing their efficiency and reliability.

Profiling and tracing integration

Thu, 07 Mar 2024 01:50:28 +0000

Profiling and tracing integration

Profiles, continuous profiling, and distributed traces are all tools that can be used to improve the performance and reliability of applications. However, each tool has its own strengths and weaknesses, and it is important to choose the right tool for the job as well as understand when to use both.

Profiling

Profiling offers a deep-dive into an application’s performance at the code level, highlighting resource usage and performance hotspots.

Usage	During development, major releases, or upon noticing performance quirks.
Benefits	Business: Boosts user experience through enhanced application performance. Technical: Gives clear insights into code performance and areas of refinement.
Example	A developer uses profiling upon noting slow app performance, identifies a CPU-heavy function, and optimizes it.

Continuous profiling

Continuous profiling provides ongoing performance insights, capturing long-term trends and intermittent issues.

Usage	Mainly in production, especially for high-priority applications.
Benefits	Business: Preemptively addresses inefficiencies, potentially saving costs. Technical: Highlights performance trends and issues like potential memory leaks over time.
Example	A month-long data from continuous profiling suggests increasing memory consumption, hinting at a memory leak.

Distributed tracing

Traces requests as they cross multiple services, revealing interactions and service dependencies.

Usage	Essential for systems like microservices where requests touch multiple services.
Benefits	Business: Faster issue resolution, reduced downtimes, and strengthened customer trust. Technical: A broad view of the system's structure, revealing bottlenecks and inter-service dependencies.
Example	In e-commerce, a user's checkout request might involve various services. Tracing depicts this route, pinpointing where time is most spent.

Combined power of tracing and profiling

When used together, tracing and profiling provide a powerful tool for understanding system and application performance.

Usage	For comprehensive system-to-code insights, especially when diagnosing complex issues spread across services and codebases.
Benefits	Business: Reduces downtime, optimizes user experience, and safeguards revenues. Technical: Holistic view: Tracing pinpoints bottle-necked services, while profiling delves into the responsible code segments. End-to-end insight: Visualizes a request's full journey and the performance of individual code parts. Efficient diagnosis: Tracing identifies service latency; profiling zeroes in on its cause, be it database queries, API calls, or specific code inefficiencies.
Example	Tracing reveals latency in a payment service. Combined with profiling, it's found that a particular function, making third-party validation calls, is the culprit. This insight guides optimization, refining system efficiency.

Profile CLI

Thu, 29 Feb 2024 16:10:16 +0000

Profile CLI

profilecli is a command-line utility that enables various productivity flows such as:

Interacting with a running Pyroscope server to upload profiles, query data, and more
Inspecting Parquet files

Hint: Use the help command (profilecli help) to get a full list of capabilities as well as additional help information.

Install Profile CLI

You can install Profile CLI using a package or by compiling the code.

Install using a package

On macOS, you can install Profile CLI using HomeBrew:

brew install pyroscope-io/brew/profilecli

For other platforms, you can manually download the profilecli release asset for your operating system and architecture and make it executable.

For example, for Linux with the AMD64 architecture:

Download and extract the package (archive).

curl -fL https://github.com/grafana/pyroscope/releases/download/v1.1.5/profilecli_1.1.5_linux_amd64.tar.gz | tar xvz

Make profilecli executable:
Bash
```
chmod +x profilecli
```
Optional: Make profilecli reachable from anywhere:
Bash
```
sudo mv profilecli /usr/local/bin
```

Build from source code

To build from source code, you must have:

Go installed (> 1.19).
Either $GOPATH or $GOBIN configured and added to your PATH environment variable.

To build the source code:

Clone the repository.

git clone git@github.com:grafana/pyroscope.git

Run the Go install command to build and install the package.
Bash
```
cd pyroscope
go install ./cmd/profilecli
```
The command places the profilecli executable in $GOPATH/bin/ (or $GOBIN/) and make it available to use.

Common flags and environment variables

The profilecli commands that interact with a Pyroscope server require a server URL and optionally authentication details. These can be provided as command-line flags or environment variables.

Server URL

default: http://localhost:4040

The --url flag specifies the server against which the command will run. If using Grafana Cloud, an example URL could be https://profiles-prod-001.grafana.net. For local instances, the URL could look like http://localhost:4040.
Authentication details.

default: <empty>

If using Grafana Cloud or authentication is enabled on your Pyroscope server, you will need to provide a username and password using the --username and --password flags respectively. For Grafana Cloud, the username will be the Stack ID and the password the generated API token.

Environment variable naming

You can use environment variables to avoid passing flags to the command every time you use it, or to protect sensitive information. Environment variables have a PROFILECLI_ prefix. Here is an example of providing the server URL and credentials for the profilecli tool:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>
# now we can run a profilecli command without specifying the url or credentials:
profilecli <command>

Uploading a profile to a Pyroscope server using `profilecli`

Using profilecli streamlines the process of uploading profiles to Pyroscope, making it a convenient alternative to manual HTTP requests.

Prerequisites

Ensure you have profilecli installed on your system by following the installation steps above.
Have a profile file ready for upload. Note that you can only upload pprof files at this time.

Upload steps

Identify the pprof file.
- Path to your pprof file: path/to/your/pprof-file.pprof
Optional: Specify any extra labels.
- You can add additional labels to your uploaded profile using the --extra-labels flag.
- You can provide the name of the application that the profile was captured from via the service_name label (defaults to profilecli-upload). This will be useful when querying the data via profilecli or the UI.
- You can use the flag multiple times to add several labels.

Construct and execute the Upload command.

Here’s a basic command template:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>

profilecli upload --extra-labels=<label_name>=<label_value> <pprof_file_path>

Example command:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli upload path/to/your/pprof-file.pprof

Example command with extra labels:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli upload \
    --extra-labels=service_name=my_application_name \
    --extra-labels=cluster=us-east \
    path/to/your/pprof-file.pprof

Check for successful upload.
- After running the command, you should see a confirmation message indicating a successful upload. If there are any issues, profilecli provides error messages to help you troubleshoot.

Querying a Pyroscope server using `profilecli`

You can use the profilecli query command to look up the available profiles on a Pyroscope server and read actual profile data. This can be useful for debugging purposes or for integrating profiling in CI pipelines (for example to facilitate profile-guided optimization).

Looking up available profiles on a Pyroscope server

You can use the profilecli query series command to look up the available profiles on a Pyroscope server. By default, it queries the last hour of data, though this can be controlled with the --from and --to flags. You can narrow the results down with the --query flag. See profilecli help query series for more information.

Query series steps

Optional: Specify a Query and a Time Range.
- You can provide a label selector using the --query flag, for example: --query='{service_name="my_application_name"}'.
- You can provide a custom time range using the --from and --to flags, for example, --from="now-3h" --to="now".

Construct and execute the Query Series command.

Here’s a basic command template:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>

profilecli query series --query='{<label_name>="<label_value>"}'

Example command:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli query series --query='{service_name="my_application_name"}'

Example output:

{
    "__name__":"memory",
    "__period_type__":"space",
    "__period_unit__":"bytes",
    "__profile_type__":"memory:inuse_objects:count:space:bytes",
    "__service_name__":"my_application_name",
    "__type__":"inuse_objects",
    "__unit__":"count",
    "cluster":"eu-west-1",
    "service_name":"my_application_name"
 }

Reading a raw profile from a Pyroscope server

You can use the profilecli query merge command to retrieve a merged (aggregated) profile from a Pyroscope server. The command merges all samples found in the profile store for the specified query and time range. By default it looks for samples within the last hour, though this can be controlled with the --from and --to flags. The source data can be narrowed down with the --query flag in the same way as with the series command.

Query merge steps

Specify optional flags.
- You can provide a label selector using the --query flag, for example, --query='{service_name="my_application_name"}'.
- You can provide a custom time range using the --from and --to flags, for example, --from="now-3h" --to="now".
- You can specify the profile type via the --profile-type flag. The available profile types are listed in the output of the profilecli query series command.

Construct and execute the Query Merge command.

Here’s a basic command template:

export PROFILECLI_URL=<pyroscope_server_url>
export PROFILECLI_USERNAME=<username>
export PROFILECLI_PASSWORD=<password>

profilecli query merge \
    --profile-type=<profile_type> \
    --query='{<label_name>="<label_value>"' \
    --from="<from>" --to="<to>"

Example command:

export PROFILECLI_URL=https://profiles-prod-001.grafana.net
export PROFILECLI_USERNAME=my_username
export PROFILECLI_PASSWORD=my_password

profilecli query merge \
    --profile-type=memory:inuse_space:bytes:space:bytes \
    --query='{service_name="my_application_name"}' \
    --from="now-1h" --to="now"

Example output:

level=info msg="query aggregated profile from profile store" url=http://localhost:4040 from=2023-12-11T13:38:33.115683-04:00 to=2023-12-11T14:38:33.115684-04:00 query={} type=memory:inuse_space:bytes:space:bytes
PeriodType: space bytes
Period: 524288
Time: 2023-12-11 13:59:59.999 -0400 AST
Duration: 59m5
Samples:
inuse_space/bytes[dflt]
  115366240: 107 13 14 15 16 17 1 2 3
...

View and analyze profile data on Grafana Labs

Understand profiling types and their uses in Pyroscope

Understand profiling types and their uses in Pyroscope

Profiling types

Available profile types

CPU profiling

Memory allocation profiling

Goroutine profiling

Mutex profiling

Block profiling

Flamegraphs: Visualizing performance data

Flamegraphs: Visualizing performance data

How is a flamegraph created?

What does a flamegraph represent?

Use the Pyroscope UI to explore profiling data

Use the Pyroscope UI to explore profiling data

Key features of the Pyroscope UI

Tag Explorer

Single view

Comparison page

Diff page: Identify changes with differential analysis

Understand 'self' vs. 'total' metrics in profiling with Pyroscope

Understand ‘self’ vs. ’total’ metrics in profiling with Pyroscope

Self

Total

‘Self’ and ‘Total’ in CPU profiling

‘Self’ and ‘Total’ in memory profiling

Conclusion

Profiling and tracing integration

Profiling and tracing integration

Profiling

Continuous profiling

Distributed tracing

Combined power of tracing and profiling

Profile CLI

Profile CLI

Install Profile CLI

Install using a package

Build from source code

Common flags and environment variables

Environment variable naming

Uploading a profile to a Pyroscope server using profilecli

Prerequisites

Upload steps

Querying a Pyroscope server using profilecli

Looking up available profiles on a Pyroscope server

Query series steps

Reading a raw profile from a Pyroscope server

Query merge steps

Uploading a profile to a Pyroscope server using `profilecli`

Querying a Pyroscope server using `profilecli`