Grafana Learn
Sign in Sign up
Path

Learn how to read a flame graph to see where a service spends its resources.

Estimated time: 2 min

Investigate with flame graphs

A flame graph is a hierarchical view of call stacks. It answers one question: where did this service spend its resources? Here that resource is CPU time, since you’re viewing a CPU profile.

Horizontally, the full width is 100% of the current view, not the whole run. That view is the profile type, service, filters, and time range you selected. Each block is a function, and its width is the share of that total it accounts for, including everything it calls. Wider costs more. Left to right isn’t a timeline: blocks aren’t ordered by when they ran.

Vertically, each level is a caller–callee relationship. The root sits at the top, and callees stack beneath their callers. Depth shows how deep the call stack goes, not how expensive it is. A tall, narrow tower is a deep path that costs little. A short, wide block costs a lot.

Self time is spent in the function itself. Total time is the function plus its callees. One function can appear as several blocks if it’s called from several places, so check the table to see its total across the profile.

This path uses CPU. You’ll look at a lower-CPU region’s flame graph first, then eu-west-1, so the difference is easier to see.

To explore flame graphs, complete the following steps:

  1. Select Flame graph for ap-south-1, a lower-CPU region.

    Take a look at how CPU time is distributed across the call stack. Keep this picture in mind as a baseline.

  2. In the Filters bar, change the region filter from ap-south-1 to eu-west-1.

  3. In the flame graph search bar, type main, regex to filter the function list.

  4. Hover over the top of the flame graph to inspect how much time this path uses.

    Nearly half the CPU time is used by the PlaceOrder function. Compared with ap-south-1, a lot more of that time is in regex-related calls.

  5. Eye-balling two flame graphs works, but there’s an easier way. Return to the Labels view with region filtered so you can open the per-region panels again.

  6. Click Select on the region panel to show each region side by side for the Diff setup.

Next, use Diff flame graph to compare the regions together.

More to explore