Systems Performance 2nd Ed.

BPF Performance Tools book

Recent posts:
Blog index

USENIX/LISA 2013 Blazing Performance with Flame Graphs

23 Apr 2017

In 2013 I gave a plenary at USENIX LISA on flame graphs: my visualization for profiled stack traces, which is now used by many companies (including Netflix, Facebook, and Linkedin) to identify which code paths consume CPU. The talk is more relevant today, now that flame graphs are widely adopted.

Here are the slides or as a PDF:

/ permalink/zoom

Video is on youtube:

The talk explains the origin of flame graphs, how to interpret them, and then tours different profile and trace event types that can be visualized. It predates some flame graph features that were added later: zoom, search, mixed-mode color highlights (--colors=java), and differential flame graphs.

I used DTrace to create different types of flame graphs in the talk, but since then I've developed ways to do them on Linux, using perf for CPU flame graphs, and bcc/BPF for advanced flame graphs: off-CPU and more. My BPF off-CPU flame graphs post used my stack track hack, but since then we've added stack trace support to BPF in Linux (4.6), and these can now be implemented without hacks. The tool offcputime in bcc has already been updated to do this (thanks Vicent Marti and others for getting it working well, and Alexei Starovoitov for adding stack trace support to BPF).

This talk was 170 slides in 90 minutes, which may have been too much in 2013 when flame graphs were new. There's a reason for this: I'd planned to do a 45 minute talk on CPU flame graphs, ending on slide 98, followed by a different talk. For reasons beyond my control, I was told the night before that I couldn't give that second talk. My plan B, as I'd already discussed with the conference organizers, was to extend the flame graphs talk and add an advanced section. I was up to 5am doing this, and was then woken at 8am by the conference organizers: the plenary speaker had shellfish poisoning, and could I come down and give my flame graphs talk at 9am, instead of later that day? That's how this ended up as a 90 minute plenary!

At that LISA I also worked more with USENIX staff, and co-delivered a metrics workshop, as well as another talk. I was proud to be involved with USENIX/LISA and contribute in these ways. And, you can too, the call for proposals for LISA 2017 ends tomorrow (April 24).

Since 2013, I've also written about flame graphs in ACMQ and CACM. For the latest on flame graphs, see the updates section of my flame graphs page.

Click here for Disqus comments (ad supported).