Hot/Cold Flame Graphs

Hot/Cold Flame Graph

This is an experimental visualization which combines both CPU and Off-CPU flame graphs. This shows all thread time in one graph, and allows direct comparisons between on- and off-CPU code path durations.

To illustrate the coverage, Figure 1 shows a generic thread state transition diagram, with the (hot) on-CPU state in red, and (cold) off-CPU states in blue:

Figure 1. Thread state transition diagram.

Having one flame graph show all of these states should be incredibly useful. However, so far these hot/cold flame graphs have proved difficult to use, and this remains experimental. I've created this page to share some work in this area.


These include:

The following sections show different possibilities for hot/cold flame graphs: separate, mixed, and integrated.

1. Separate

One approach to solve some of these difficulties is also the easiest: simply generate separate on- and off-CPU flame graphs and show them together:

This solves the compressed on-CPU time problem, and also to some degree avoids confusion from mixing the types under one graph.

2. Mixed

The hot/cold flames can be shown together, scaled to the same x-axis. The following example shows a single kernel thread, with on- and off-CPU paths (SVG, PNG; sorry, this is an older version that lacks zoom):

I shared the code for generating this under the dev directory of FlameGraph tools. As mentioned under Difficulties, the relatively large amount of off-CPU time can squeeze the on-CPU time. Showing per-thread hot/cold flame graphs helps to a point.

3. Integrated

Vladimir Kirillov has included blocking calls in eflame, an Erlang Flame Graph profiler, by integrating the blocking data with the CPU profile, so that ancestry can be merged. The final blocking function is shown as blue on top of a warm stack. One of his examples:

It looks effective to me, and keeps the ancestry stacks as merged as possible. Although, this does diminish the visual effect of the blue color for blocking. Perhaps this is still the best approach so far?


Bruce Dawson also suggested this type of visualization while he was working with Xperf data; see the comment thread.

See the main Flame Graphs page for other types of flame graphs and links, and the flame graph software.

Last updated: 19-Jan-2016