Grafana Learn
Sign in Sign up
Path

Compare two regions with Diff flame graph to find the function responsible for higher CPU usage.

Estimated time: 2 min

Compare with Diff flame graph

Now compare the high-CPU region (eu-west-1) to a lower-CPU one (ap-south-1) with Diff flame graph. Put the behavior you expect on Baseline and what you’re investigating on Comparison.

The view merges both profiles. It compares each function’s share of each side’s total time, not absolute time, so the two sides stay comparable even if they did different amounts of work. Width is both profiles combined; color is what changed. Red is a larger share in the comparison, green a smaller share (blue in the colorblind scheme), and gray little or no change. Searching also grays out functions that don’t match. Stronger color means a bigger relative change.

Hover a function or read the table for Baseline and Comparison as % of each side’s total. In the table, Diff is the relative change between them; it reads new when the function is absent from the baseline and removed when it’s absent from the comparison.

You can also open Diff flame graph from the tabs and set region filters yourself. Here, you’ll select them on the region panels instead.

To use Diff flame graph, complete the following steps:

  1. Return to the Labels view with region filtered. If you already see the region panel with Select, or the per-region panels, click Skip.

  2. Click Select on the region panel so each region has its own panel. If the per-region panels are already open, click Skip.

  3. Assign the high-CPU region as Comparison and a lower-CPU region as Baseline.

    1. On the eu-west-1 panel, select Comparison.

    2. On the ap-south-1 panel, select Baseline.

  4. Click Compare to open the Diff flame graph view for the two regions.

  5. You can click and drag on the Baseline and Comparison graphs to highlight the time ranges you want to diff. For this path, click Auto-select to pick windows for you.

  6. In the table search bar, type main to filter the function list.

  7. You can see that PlaceOrder calls processOrder, which calls validateOrderWithRegexp. Hover over validateOrderWithRegexp and check its Baseline and Comparison percentages. ap-south-1 spends no time there, so Baseline is 0% and the table marks the function new. That regex work exists only in eu-west-1, which confirms it as the cause of the increased CPU usage.

You’ve found the function responsible for the higher CPU usage in eu-west-1. Next is an optional AI look at Diff flame graph.