Configure the client to send profiles on Grafana Labs

Profile types and instrumentation

Wed, 08 Apr 2026 14:38:28 +0000

Profile types and instrumentation

Profiling is an essential tool for understanding and optimizing application performance. In Pyroscope, various profiling types allow for an in-depth analysis of different aspects of your application. This guide explores these types and explain their impact on your program.

Profiling types refer to different dimensions of application performance analysis, focusing on specific aspects like CPU usage, memory allocation, or thread synchronization.

Note that when Pyroscope receives a Java wall profile, both cpu and wall profiles are automatically ingested, even if cpu profiling is turned off.

Pyroscope supports these profile types:

CPU (CPU time, wall time)
Memory (allocation objects, allocation space, heap)
In use objects and in-use space
Goroutines
Mutex count and duration
Block count and duration
Lock count and duration
Exceptions

Refer to Understand profiling types and their uses in Pyroscope for more details about the profile types.

Profile type support by instrumentation method

The instrumentation method you use determines which profile types are available. You can use either auto or manual instrumentation.

Auto-instrumentation with Grafana Alloy

You can send data from your application using Grafana Alloy collector. Alloy supports profiling with eBPF, Java, and Golang in pull mode.

The eBPF profiler collects CPU profiles. Natively compiled languages like C/C++, Go, Rust, and Zig are supported. Frame pointers are not required — the profiler uses .eh_frame data for unwinding. Refer to Troubleshooting unknown symbols for additional requirements and information.

The following high-level languages are also supported: Java (Hotspot JVM), .NET, Python, Ruby, PHP, Node.js, and Perl. Each high-level language can be individually enabled or disabled in the pyroscope.ebpf Alloy component configuration.

For more information, refer to Configure the client to send profiles with Grafana Alloy.

This table lists the available profile types based on auto instrumentation using Alloy.

Profile type	Go (pull)	Java	eBPF
CPU	Yes	Yes	Yes
Alloc Objects	Yes	Yes
Alloc Space	Yes	Yes
Inuse Objects
Inuse Space
Goroutines	Yes
Mutex Count
Mutex Duration
Block Count	Yes
Block Duration	Yes
Lock Count		Yes
Lock Duration		Yes
Exceptions
Wall		Yes
Heap

Instrumentation with SDKs

Using the Pyroscope language SDKs lets you instrument your application directly for precise profiling. You can customize the profiling process according to your application’s specific requirements.

For more information on the language SDKs, refer to Pyroscope language SDKs.

This table lists the available profile types based on the language SDK.

Profile type	Go (push)	Java	.NET	Ruby	Python	Rust	Node.js
CPU	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Alloc Objects	Yes	Yes	Yes
Alloc Space	Yes	Yes	Yes
Inuse Objects	Yes		Yes (7.0+)
Inuse Space	Yes		Yes (7.0+)
Goroutines	Yes
Mutex Count	Yes		Yes
Mutex Duration	Yes		Yes
Block Count	Yes
Block Duration	Yes
Lock Count		Yes	Yes
Lock Duration		Yes	Yes
Exceptions			Yes
Wall		Yes	Yes				Yes
Heap			Yes (7.0+)				Yes

Profile types supported with span profiles

Pyroscope can integrate with distributed tracing systems supporting the OpenTelemetry standard. This integration lets you link traces with the profiling data and find resource usage for specific lines of code for your trace spans.

Only CPU profile type is supported for span profiles.

The following languages are supported:

OpenTelemetry profiling support

Wed, 08 Apr 2026 14:38:28 +0000

OpenTelemetry profiling support

Pyroscope includes experimental support for receiving and visualizing profiles from OpenTelemetry sources. This integration allows you to:

Collect system-wide profiles using the OpenTelemetry eBPF profiler
Process profile data through the OpenTelemetry Collector
Visualize profiles in Grafana using Pyroscope

Considerations

Before getting started, you should consider the following limitations:

The OpenTelemetry profiles protocol (proto files) is under active development:
- Breaking changes are expected and have occurred
- Compatibility between components (profiler, collector, backend) requires careful version management
- We maintain support for the latest protocol version, but updates may be required frequently
Symbolization support is currently limited:
- Function names may not appear in flamegraphs for some programs
- We’re working on improving symbol resolution and adding support for manual symbol uploads
We recommend evaluating this feature for development and testing purposes, but waiting for protocol stabilization before production use

Requirements

Linux system (amd64/arm64) for eBPF profiler
OpenTelemetry Collector with profiles feature gate enabled
Grafana with Pyroscope data source enabled

Architecture

The profile collection pipeline consists of:

OpenTelemetry eBPF Profiler: Collects system-wide profiles
OpenTelemetry Collector: Receives and forwards profile data
Pyroscope: Stores and processes profiles
Grafana: Visualizes profile data

Get started

For detailed setup instructions and working examples, refer to the examples repository.

The examples demonstrate deployments for both Docker and Kubernetes environments.

Grafana Alloy

Wed, 08 Apr 2026 14:38:28 +0000

Grafana Alloy

You can send data from your application using Grafana Alloy (preferred) or Grafana Agent (legacy) collectors. Both collectors support profiling with eBPF, Java, and Golang in pull mode.

Grafana Alloy is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector. Alloy uniquely combines the very best OSS observability signals in the community. Alloy uses configuration files written in Alloy configuration syntax. For more information, refer to the Alloy configuration syntax.

Alloy is the recommended collector instead of Grafana Agent. New installations should use Alloy.

The instructions in this section explain how to use Alloy.

Note
Refer to Available profiling types for a list of supported profile types.

Legacy collector, Grafana Agent

Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Grafana Agent is a legacy tool for collecting and forwarding profiling data. Agent supports for eBPF and Golang in pull mode. For information about Agent, refer to Grafana Agent Flow.

Instructions for using Grafana Agent are available in documentation for Pyroscope v1.8 and earlier.

eBPF profiling

eBPF (Extended Berkeley Packet Filter) is a modern Linux kernel technology that allows for safe, efficient, and customizable tracing of system and application behaviors without modifying the source code or restarting processes.

Benefits of eBPF profiling:

Low overhead: eBPF collects data with minimal impact on performance.
Versatile: eBPF can trace system calls, network packets, and even user-space application logic.
Dynamic: No need to recompile or restart applications. eBPF allows for live tracing.

Set up eBPF profiling

Ensure your system runs a Linux kernel version 4.9 or newer.
Install a collector, such as Alloy, on the target machine or container.
Configure Alloy to use eBPF for profiling. Refer to the eBPF documentation for detailed steps.
The collector collects eBPF profiles and sends them to the Pyroscope server.

Supported languages

Golang profiling in pull mode

In pull mode, the collector periodically retrieves profiles from Golang applications, specifically targeting the pprof endpoints.

Benefits of Golang profiling in pull mode

Non-intrusive: No need to modify your application’s source code.
Centralized profiling: Suitable for environments with multiple Golang applications or microservices.
Automatic: Alloy handles the pulling and sending of profiles, requiring minimal configuration.

Set up Golang profiling in pull mode

Ensure your Golang application exposes pprof endpoints.
Install and configure Alloy on the same machine or container where your application runs.
Ensure Alloy is set to pull mode and targeting the correct pprof endpoints. For step-by-step instructions, visit the Go (Pull Mode) documentation.
The collector queries the pprof endpoints of your Golang application, collects the profiles, and forwards them to the Pyroscope server.

Receive profiles from Pyroscope SDKs

Alloy can receive profiles from applications instrumented with Pyroscope SDKs through the pyroscope.receive_http component. This approach provides several key advantages:

Improved performance by sending profiles to a local Alloy instance instead of over the internet to Grafana Cloud, reducing latency and application impact
Separation of infrastructure concerns from application code - developers don’t need to handle authentication, tenant configuration, or infrastructure labels in their code
Centralized management of authentication and metadata enrichment (for example, Kubernetes labels, business labels)

This capability is not available in the legacy Grafana Agent.

Set up profile receiving

Configure your application with a Pyroscope SDK pointing to the pyroscope.receive_http Alloy component.
For step-by-step instructions, refer to the Receive profiles from Pyroscope SDKs documentation.

Next steps

Whether using eBPF for versatile system and application profiling or relying on Golang’s built-in pprof endpoints in pull mode, Alloy collectors offer streamlined processes to gather essential profiling data. Choose the method that best fits your application and infrastructure needs.

Pyroscope language SDKs

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope language SDKs

The following languages SDKs provide support for sending profiles from your application:

Golang Documentation Examples	Java Documentation Examples	Python Documentation Examples	Ruby Documentation Examples
Node.js Documentation Examples	.NET Documentation Examples	eBPF Documentation Examples	Rust Documentation Examples

Note
Refer to Available profiling types for a list of profile types supported by each language.

If you’re interested in integrating other ecosystems, reach out to us on GitHub.

Link tracing and profiling with Span Profiles

Wed, 08 Apr 2026 14:38:28 +0000

Link tracing and profiling with Span Profiles

Span Profiles are a powerful feature that further enhances the value of continuous profiling. Span Profiles offer a novel approach to profiling by providing detailed insights into specific execution scopes of applications, moving beyond the traditional system-wide analysis to offer a more dynamic, focused analysis of individual requests or trace spans.

This method enhances understanding of application behavior by directly linking traces with profiling data, enabling engineering teams to pinpoint and resolve performance bottlenecks with precision.

Key benefits and features:

Deep analysis: Understand the specifics of code execution within particular time frames, offering granular insights into application performance
Seamless integration: Smoothly transition from a high-level trace overview to detailed profiling of specific trace spans within Grafana’s trace view
Efficiency and cost savings: Quickly identify and address performance issues, reducing troubleshooting time and operational costs

Note
Span profiling is only effective on spans longer than 20ms to ensure statistical accuracy.

Get started

Select an option from the list below:

Configure Pyroscope: Begin sending profiling data to unlock the full potential of Span Profiles
Client-side packages: Easily link traces and profiles using available packages for Go, Java, Ruby, .NET, and Python
Configure the Tempo data source in Grafana or Grafana Cloud to discover linked traces and profiles.

To learn more, check out the product announcement blog: Introducing Span Profiles.

AWS Lambda profiling extension

Wed, 08 Apr 2026 14:38:28 +0000

AWS Lambda profiling extension

The Pyroscope AWS Lambda extension is a robust tool for profiling AWS Lambda functions, ensuring minimal latency impact. This profiling is essential for optimizing your functions.

Why profile AWS Lambda functions?

AWS Lambda functions, while powerful and flexible, can lead to significant costs if not managed efficiently. Serverless architectures, like AWS Lambda, can mask performance issues. Since Lambda functions are billed based on execution time and allocated memory, code inefficiencies can lead to higher costs. Often, these costs accumulate unnoticed because of the following reasons:

Granular billing: Lambda functions are billed in milliseconds, which can make small inefficiencies seem insignificant at first. However, when scaled to thousands or millions of invocations, these inefficiencies can lead to substantial costs.
Complex performance profile: Lambda functions may interact with various services and resources, making it challenging to pinpoint performance bottlenecks.
Variable load: The serverless nature of AWS Lambda means that functions might handle variable loads at different times, making it hard to optimize for every scenario.

Profiling Lambda functions helps identify these hidden performance bottlenecks, enabling developers to optimize their code for both performance and cost. Effective profiling can reveal inefficient code paths, unnecessary memory usage, and areas where the execution time can be reduced. By addressing these issues, organizations can significantly reduce their AWS bill, improve application responsiveness, and ensure a more efficient use of resources.

Architecture

This extension runs a relay server on the same network namespace as the Lambda function, ensuring minimal added latency.

For more details, refer to the Building Extensions for AWS Lambda blog post.

Set up the Pyroscope Lambda extension

To set up the Pyroscope Lambda extension, you need to:

Configure your Lambda function
Set up your environment variables
Integrate the Pyroscope SDK

Configure your Lambda function

Configure your Lambda function to use the extension. Find the latest release on our releases page.

Set up the environment variables

Configure the extension with the following environment variables:

Environment Variable	Default Value	Description
`PYROSCOPE_REMOTE_ADDRESS`	`https://profiles-prod-001.grafana.net`	Destination for relayed Pyroscope data
`PYROSCOPE_BASIC_AUTH_USER`	`""`	HTTP Basic authentication user
`PYROSCOPE_BASIC_AUTH_PASSWORD`	`""`	HTTP Basic authentication password
`PYROSCOPE_SELF_PROFILING`	`false`	Whether to profile the extension itself
`PYROSCOPE_LOG_LEVEL`	`info`	Log level (`error`, `info`, `debug`, `trace`)
`PYROSCOPE_TIMEOUT`	`10s`	HTTP client timeout (in Go duration format)
`PYROSCOPE_NUM_WORKERS`	`5`	Number of relay workers
`PYROSCOPE_TENANT_ID`	`""`	Pyroscope tenant ID (for multi-tenancy)

Integrate the Pyroscope SDK

The Pyroscope AWS Lambda extension is compatible with all existing Pyroscope SDKs. Here are some key considerations:

Initialize the SDK before setting up the AWS Lambda handler.
Ensure that the Pyroscope server address is configured to http://localhost:4040.

Note that the SDK packages are not automatically included in the extension layer. For Java, Python, Node.js, and Ruby, you must either include the SDK package in the function deployment package or add it as a Lambda layer. Refer to the detailed guide in the AWS Lambda documentation for your specific runtime for further instructions:

For a Golang Lambda function, integrate the Pyroscope SDK as follows:

func HandleRequest(ctx context.Context) (string, error) {
    return "Hello world!", nil
}

func main() {
    pyroscope.Start(pyroscope.Config{
        ApplicationName: "simple.golang.lambda",
        ServerAddress:   "http://localhost:4040",
    })
    lambda.Start(HandleRequest)
}

Replace simple.golang.lambda with your application name.

Use cases

Once set up, you can use the Pyroscope UI to analyze your Lambda function’s data to facilitate performance optimizations. For more on this, visit our Pyroscope AWS Lambda Extension blog post.

Send data to Pyroscope

To configure the extension for data transmission:

PYROSCOPE_REMOTE_ADDRESS="<URL>"
PYROSCOPE_BASIC_AUTH_USER="<User>"
PYROSCOPE_BASIC_AUTH_PASSWORD="<Password>"
# PYROSCOPE_TENANT_ID="<TenantID>" # For multi-tenant mode

Replace placeholders accordingly. For sending data to Grafana, use your Grafana stack user and API key for authentication.

Pyroscope memory overhead

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope memory overhead

Pyroscope has very low memory overhead, usually less than 50 MB per pod.

How the profiler works

Stacktraces are captured at regular intervals (~100Hz). Memory allocations and lock contention events are sampled.

These stacktraces and memory allocation events are temporarily stored in memory.

The stored profiling data is periodically (default is every 15 seconds) sent to the server.

Memory overhead consists of:

The temporary storage of profiles in memory is the primary contributor to memory overhead.
Memory usage typically scales up sublinearly with the number of CPUs.

What happens if the Pyroscope backend is down?

The guiding principle of Pyroscope clients is to never cause the user application to crash.

Profiles are uploaded using multiple threads. The default value is 5 and can be adjusted using the Threads variable.

If the backend is down or slow, the profiler discards new profiles to prevent running out of memory.

Real-world example

The exact overhead can vary based on the application, so direct testing is recommended.

At Grafana Labs we continuously profile all of our workloads. With all profiling types enabled (CPU, alloc, goroutine, mutex, block), the observed memory overhead per pod is typically less than 50 MB.

The overhead is often so minimal that it becomes challenging to accurately measure.