Introduction on Grafana Labs

What is profiling?

Wed, 08 Apr 2026 14:38:28 +0000

What is profiling?

Profiling is a technique used in software development to measure and analyze the runtime behavior of a program. By profiling a program, developers can identify which parts of the program consume the most resources, such as CPU time, memory, or I/O operations. You can use this information to optimize the program, making it run faster or use fewer resources.

Pyroscope can be used for both traditional and continuous profiling.

Traditional profiling (non-continuous)

Traditional profiling, often referred to as sample-based or instrumentation-based profiling, has its roots in the early days of computing. Back then, the primary challenge was understanding how a program utilized the limited computational resources available.

With sample-based profiling, the profiler interrupts the program at regular intervals, capturing the program’s state each time. By analyzing these snapshots, developers can deduce the frequency at which parts of the code execute.

With instrumentation-based profiling, developers insert additional code into the program that records information about its execution. This approach provides detailed insights but can alter the program’s behavior due to the added code overhead.

Benefits

Traditional profiling provides:

Precision: Offers a deep dive into specific sections of the code.
Control: Developers can initiate profiling sessions at their discretion, allowing for targeted optimization efforts.
Detailed reports: Provides granular data about program execution, making it easier to pinpoint bottlenecks.

Continuous profiling

As software systems grew in complexity and scale, the limitations of traditional profiling became evident. Issues could arise in production that weren’t apparent during limited profiling sessions in the development or staging environments.

This led to the development of continuous profiling, a method where the profiling data is continuously collected in the background with minimal overhead. By doing so, developers gain a more comprehensive view of a program’s behavior over time, helping to identify sporadic or long-term performance issues.

Benefits

Continuous profiling provides:

Consistent monitoring: Unlike traditional methods that offer snapshots, continuous profiling maintains an uninterrupted view, exposing both immediate and long-term performance issues.
Proactive bottleneck detection: By consistently capturing data, performance bottlenecks are identified and addressed before they escalate, reducing system downtime and ensuring smoother operations.
Broad performance landscape: Provides insights across various platforms, from varied technology stacks to different operating systems, ensuring comprehensive coverage.
Bridging the Dev-Prod gap: Continuous profiling excels in highlighting differences between development and production:
- Hardware discrepancies: Unearths issues stemming from differences in machine specifications.
- Software inconsistencies: Sheds light on variations in software components that might affect performance.
- Real-world workload challenges: Highlights potential pitfalls when real user interactions and loads don’t align with development simulations.
Economical advantages:
- Resource optimization: Continual monitoring ensures resources aren’t wasted, leading to cost savings.
- Rapid problem resolution: Faster troubleshooting means reduced time and monetary investment in issue rectification, letting developers channel their efforts into productive endeavors.
Non-intrusive operation: Specifically designed to work quietly in the background, continuous profiling doesn’t compromise the performance of live environments.
Real-time response: It equips teams with the ability to act instantly, addressing issues as they arise rather than post-occurrence, which is crucial for maintaining high system availability.

Choose between traditional and continuous profiling

In many modern development workflows, both methods are useful.

	Traditional profiling	Continuous profiling
When to use	During development or testing phases	In production environments or during extended performance tests.
Advantages	Offers detailed insights that can target specific parts of code.	Provides a continuous view of system performance, often with minimal overhead, making it suitable for live environments.
Disadvantages	Higher overhead provides only a snapshot in time.	It might be less detailed than traditional profiling due to the need to minimize impact on the running system.

What is continuous profiling?

Wed, 08 Apr 2026 14:38:28 +0000

What is continuous profiling?

Continuous profiling is a systematic method of collecting and analyzing performance data from production systems.

Traditionally, profiling is used to debug applications on an as-needed basis. For example, you can run a benchmark tool locally and get a pprof file in Go or connect to a misbehaving prod instance and pull a flame graph from a JFR file in Java. This method is good for debugging, but not robust enough for production.

Refer to Flame graphs to learn more.

Continuous profiling is a modern approach which is safer and more scalable for production environments. It uses low-overhead sampling to collect profiles from production systems and stores the profiles in a database for later analysis. Using continuous profiling gives you a more holistic view of your application and how it behaves in production.

Grafana offers Grafana Pyroscope and Grafana Cloud Profiles (powered by Pyroscope) to collect and store your profiling data. You can use Grafana Profiles Drilldown to inspect profile data and investigate issues.

Benefits

Why prioritize continuous profiling?

In-depth code insights: It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance.
Complements other observability tools: Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy.
Proactive performance optimization: Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications.

Use cases

Adopting continuous profiling with tools like Grafana Pyroscope and Profiles Drilldown can lead to significant business advantages:

Reduced operational costs: Optimization of resource usage can significantly cut down cloud and infrastructure expenses
Reduced latency: Identifying and addressing performance bottlenecks leads to faster and more efficient applications
Enhanced incident management: Faster problem identification and resolution, reducing Mean Time to Resolution (MTTR) and improving end-user experience

Reduced operational costs

By providing in-depth insights into application performance, profiling empowers teams to identify and eliminate inefficiencies, leading to significant savings in areas like observability, incident management, messaging/queuing, deployment tools, and infrastructure.

By using sampling profilers, Pyroscope and Cloud Profiles can collect data with minimal overhead (~2-5% depending on a few factors). The custom storage engine compresses and stores the data efficiently. Some advantages of this are:

Low CPU overhead thanks to sampling profiler technology

Control over profiling data granularity (10s to multiple years)
Efficient compression, low disk space requirements and cost

Reduced latency

Profiles play a pivotal role in reducing application latency by identifying performance bottlenecks at the code level. This granular insight allows for targeted optimization, leading to faster application response times, improved user experience, and consequently, better business outcomes like increased customer satisfaction and revenue.

Enhanced incident management

Pyroscope and Profiles Drilldown streamline incident management by offering immediate, actionable insights into application performance issues. With continuous profiling, teams can quickly pinpoint the root cause of an incident, reducing the mean time to resolution (MTTR) and enhancing overall system reliability and user satisfaction.

Profiling types and their uses

Wed, 08 Apr 2026 14:38:28 +0000

Profiling types and their uses

Profiling is an essential tool for understanding and optimizing application performance. In Grafana Pyroscope, various profiling types allow for an in-depth analysis of different aspects of your application.

Profiling types

Profiling types refer to different dimensions of application performance analysis, focusing on specific aspects like CPU usage, memory allocation, or thread synchronization.

Pyroscope supports these profile types:

CPU (CPU time, wall time)
Memory (allocation objects, allocation space, heap)
In use objects and in-use space
Goroutines
Mutex count and duration
Block count and duration
Lock count and duration
Exceptions

Refer to the Profile types tables for information on supported profile types based on instrumentation method.

For information on auto-instrumentation and supported language SDKs, refer to Configure the client.

CPU profiling

CPU profiling measures the amount of CPU time consumed by different parts of your application code. High CPU usage can indicate inefficient code, leading to poor performance and increased operational costs. It’s used to identify and optimize CPU-intensive functions in your application.

When to use: To identify and optimize CPU-intensive functions
Flame graph insight: The width of blocks indicates the CPU time consumed by each function

The UI shows a spike in CPU along with the flame graph associated with that spike. You may get similar insights from metrics, however, with profiling, you have more details into the specific cause of a spike in CPU usage at the line level.

Memory allocation profiling

Memory allocation profiling tracks the amount and frequency of memory allocations by the application. Excessive or inefficient memory allocation can lead to memory leaks and high garbage collection overhead, impacting application efficiency.

Types: Alloc Objects, Alloc Space

When to use: For identifying and optimizing memory usage patterns
Flame graph insight: Highlights functions where memory allocation is high

The timeline shows memory allocations over time and is great for debugging memory related issues. A common example is when a memory leak is created due to improper handling of memory in a function. This can be identified by looking at the timeline and seeing a gradual increase in memory allocations that never goes down. This is a clear indicator of a memory leak.

Without profiling, this may be something that’s exhibited in metrics or out-of-memory errors (OOM) logs but with profiling you have more details into the specific function that’s allocating the memory which is causing the leak at the line level.

Goroutine profiling

Goroutines are lightweight threads in Go, used for concurrent operations. Goroutine profiling measures the usage and performance of these threads. Poor management can lead to issues like deadlocks and excessive resource usage.

When to use: Especially useful in Go applications for concurrency management
Flame graph insight: Provides a view of goroutine distribution and issues

Mutex profiling

Mutex profiling involves analyzing mutex (mutual exclusion) locks, used to prevent simultaneous access to shared resources. Excessive or long-duration mutex locks can cause delays and reduced application throughput.

Types: Mutex Count, Mutex Duration
When to use: To optimize thread synchronization and reduce lock contention
Flame graph insight: Shows frequency and duration of mutex operations

Block profiling

Block profiling measures the frequency and duration of blocking operations, where a thread is paused or delayed. Blocking can significantly slow down application processes, leading to performance bottlenecks.

Types: Block Count, Block Duration
When to use: To identify and reduce blocking delays
Flame graph insight: Identifies where and how long threads are blocked

Flame graphs

Wed, 08 Apr 2026 14:38:28 +0000

Flame graphs

Flame graphs provide a visual summary of your profile data. A flame graph is a complete visualization of hierarchical data, for example stack trace and, file system contents, with a metric, typically resource usage, attached to the data.

A fundamental aspect of continuous profiling is the flame graph, a convenient way to visualize performance data. These graphs provide a clear, intuitive understanding of resource allocation and bottlenecks within the application.

Brendan Gregg, the creator of flame graphs, was inspired by the inability to view, read, and understand stack traces using the regular profilers to debug performance issues.

How Pyroscope creates flame graphs

This diagram shows how code is turned into a flame graph. In this case, Pyroscope samples the stacktrace of your application to understand how many CPU cycles are spent in each function. It then aggregates this data and turns it into a flame graph.

What does a flame graph represent?

Horizontally, the flame graph represents 100% of the time that this application was running. The width of each node represents the amount of time spent in that function. The wider the node, the more time spent in that function. The narrower the node, the less time spent in that function.

Vertically, the nodes in the flame graph represent the hierarchy of functions called and time spent in each function. The top node is the root node and represents the total amount of time spent in the application. The nodes below it represent the functions called and time spent in each function. The nodes below those represent the functions called from those functions and time spent in each function. This continues until you reach the bottom of the flame graph.

This is a CPU profile, but profiles can represent many other types of resource such as memory, network, disk, etc.

Flame graph visualization panel UI

To learn more about the flame graph user interface in Grafana, Grafana Cloud, and Grafana Profiles Drilldown, refer to Flame graph visualization panel.

To learn more about the flame graph in the Pyroscope UI, refer to Pyroscope UI.

Profiling and tracing integration

Wed, 08 Apr 2026 14:38:28 +0000

Profiling and tracing integration

Profiles, continuous profiling, and distributed traces are all tools that can be used to improve the performance and reliability of applications. However, each tool has its own strengths and weaknesses, and it is important to choose the right tool for the job as well as understand when to use both.

Profiling

Profiling offers a deep-dive into an application’s performance at the code level, highlighting resource usage and performance hotspots.

Usage	During development, major releases, or upon noticing performance quirks.
Benefits	Business: Boosts user experience through enhanced application performance. Technical: Gives clear insights into code performance and areas of refinement.
Example	A developer uses profiling upon noting slow app performance, identifies a CPU-heavy function, and optimizes it.

Continuous profiling

Continuous profiling provides ongoing performance insights, capturing long-term trends and intermittent issues.

Usage	Mainly in production, especially for high-priority applications.
Benefits	Business: Preemptively addresses inefficiencies, potentially saving costs. Technical: Highlights performance trends and issues like potential memory leaks over time.
Example	A month-long data from continuous profiling suggests increasing memory consumption, hinting at a memory leak.

Distributed tracing

Traces requests as they cross multiple services, revealing interactions and service dependencies.

Usage	Essential for systems like microservices where requests touch multiple services.
Benefits	Business: Faster issue resolution, reduced downtimes, and strengthened customer trust. Technical: A broad view of the system's structure, revealing bottlenecks and inter-service dependencies.
Example	In e-commerce, a user's checkout request might involve various services. Tracing depicts this route, pinpointing where time is most spent.

Combined power of tracing and profiling

When used together, tracing and profiling provide a powerful tool for understanding system and application performance.

Usage	For comprehensive system-to-code insights, especially when diagnosing complex issues spread across services and codebases.
Benefits	Business: Reduces downtime, optimizes user experience, and safeguards revenues. Technical: Holistic view: Tracing pinpoints bottle-necked services, while profiling delves into the responsible code segments. End-to-end insight: Visualizes a request's full journey and the performance of individual code parts. Efficient diagnosis: Tracing identifies service latency; profiling zeroes in on its cause, be it database queries, API calls, or specific code inefficiencies.
Example	Tracing reveals latency in a payment service. Combined with profiling, it's found that a particular function, making third-party validation calls, is the culprit. This insight guides optimization, refining system efficiency.

Pyroscope and profiling in Grafana

Wed, 08 Apr 2026 14:38:28 +0000

Pyroscope and profiling in Grafana

Grafana Pyroscope can be used alongside other Grafana products such as Loki, Tempo, Mimir, and k6. You can use Pyroscope to get the most granular insight into your application and how you can use it to fix issues that you may have identified using metrics, logs, traces, or anything else.

Within Grafana and Grafana Cloud, you can use the Pyroscope data source to query and visualize profile data from within Grafana.

Visualize traces and profiles data

Here is a screenshot of the Explore page that combines traces and profiles to be able to see granular line-level detail when available for a trace span. This lets you see the exact function that’s causing a bottleneck in your application as well as a specific request.

Integrate profiles into dashboards

Here is an example of how you can integrate profiles into your dashboards. In this case, the screenshot shows memory profiles alongside panels for logs and metrics to be able to debug OOM errors alongside the associated logs and metrics.