Improved Differential Flame Graph

Rust►Flamegraph Comments

The Differential Flame Graph was introduced by Brendan Gregg in FlameGraph toolkit , which can let us debug the performance regressions. But it also has its known issue.

“If code paths vanish completely in the second profile, then there’s nothing to color blue.”

For me there is another issue.

The width of frame doesn’t present the REAL difference between two flame graphs.

Let’s see an example🌰.

Assume we have two folded files like below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# before.folded
_start;main;func1  5
_start;main;func2  10
_start;main;func4;func3  8
_start;main;func4  25
_start;main;func5;func6  15
_start;main;func5  6

# after.folded
_start;main;func2  35
_start;main;func4  0
_start;main;func4;func3  18
_start;main;func5;func6  5
_start;main;func5  20
_start;main;func7  7

To get a clear view for function changes, we can put it in a table like below:

NameBeforeAfterDelta
func150-5
func21035+25
func3818+10
func4250-25
func5620+14
func6155-10
func707+7
Total6986|96|

Original Differential Flame Graph

image-20210522103829287

  1. func1 was totally lost. Although we can use Negation to get another differential flame graph.

  2. func3 have +10 delta and func4 have down to zero with -25 delta. But they share the SAME frame width because the diff-graph is just based on after.folded.

Differential Flame Graph With Delta Width

I changed the way calculate diff-frame width which is based on Delta column in the above table, cause it’s REAL differentials. Here is the screenshot.

image-20210522123753074

  • The width of frame is based on delta, not any one of samples. So the width percentage is stand for: How much does it effects in all changes. Now, func1 shows up and func4 has outstanding optimization from the view.

  • The frame color is calculated by its OWN delta. So it means: How does the frame’s SELF samples(time) changes.

image-20210522124723998

  • For the function has not real samples, I put the total delta width on the hint message.

Try it

You can try the improved differential flame graph at my fork from the Rust🦀 version of flame graph toolkit inferno(which is 💥20x faster than Perl version).