CPU flame graphs are a useful visualization application stack traces, allowing you to quickly identify and quantify what to tune to improve performance. For Node.js they have solved countless problems on systems which have DTrace for sampling stack traces. But what about Linux?
At Netflix we have node.js in production at scale, on Linux instances in AWS EC2, and we create flame graphs using Linux perf_events and v8's --perf_basic_prof option (also works as --perf-basic-prof). In this quick blog post, I'll share how it works and how you can do it, and what needs to be fixed to improve it further.
1. The problem
Using perf_events to profile CPU usage on node.js 0.10.23:
It's interactive: mouse-over elements for details, and click the SVG to zoom. The CPU flame graphs page explains how to interpret these, and this was created using the instructions in the Linux perf section.
2. Linux perf_events JIT support
In 2009, Linux perf_events added JIT symbol support, so that symbols from language virtual machines like the JVM could be inspected. It works in the following amazingly simple way:
- Your JIT application must be modified to create a /tmp/perf-PID.map file, which is a simple text database containing symbol addresses (in hex), sizes, and symbol names.
- That's it.
perf already looks for the /tmp/perf-PID.map file, and if it finds it, it uses it for symbol translations. So only v8 needed to be modified.
3. v8 --perf-basic-prof support
In November 2013, v8 added perf_events support, enabled using the --perf-basic-prof option. This made it into node v0.11.13. It works like this:
# ~/node-v0.11.13-linux-x64/bin/node --perf-basic-prof hello.js &  31441 # ls -l /tmp/perf-31441.map -rw-r--r-- 1 root root 81920 Sep 17 20:41 /tmp/perf-31441.map # tail /tmp/perf-31441.map 14cec4db98a0 f Stub:BinaryOpICWithAllocationSiteStub(ADD_CreateAllocationMementos:String*Generic->String) 14cec4db9920 f Stub:BinaryOpICWithAllocationSiteStub(ADD_CreateAllocationMementos:String*String->String) 14cec4db99a0 f Stub:BinaryOpICWithAllocationSiteStub(ADD_CreateAllocationMementos:String*Smi->String) 14cec4db9a20 22c LazyCompile:~nextTick node.js:389 14cec4db9cc0 156 Stub:KeyedLoadElementStub 14cec4db9e80 22 KeyedLoadIC: 14cec4db9f20 22 KeyedLoadIC: 14cec4db9fc0 56 Stub:DoubleToIStub 14cec4dba080 10c Stub:KeyedStoreElementStub
This text file is what perf_events reads.
4. node.js Flame Graphs
Now that we have node 0.11.13+ running with --perf-basic-prof, we can create a flame graph using:
$ sudo bash # perf record -F 99 -p `pgrep -n node` -g -- sleep 30 # perf script > out.nodestacks01 # git clone --depth 1 http://github.com/brendangregg/FlameGraph # cd FlameGraph # ./stackcollapse-perf.pl < ../out.nodestacks01 | ./flamegraph.pl > ../out.nodestacks01.svg
You can also use stackvis, by Dave Pacheco, a node.js implementation which has extra features.
Here's an example result:
Thanks to Trevor Norris for first posting the instructions for doing this in a short gist, which you may find useful to read. He also provides a script to facilitate this.
WARNING: map file growth
We can currently only use --perf-basic-prof for short periods (hours), due to bug 3453: the perf.map file can grow endlessly, eating Gbytes in a few days. It looks like symbols are moving location (they are supposed to stay put with --perf-basic-prof), causing the map file to keep growing.
UPDATE (2016): A new option, --perf_basic_prof_only_functions (or --perf-basic-prof-only-functions) was introduced to address this bug by only logging interesting types of symbols, cutting down on map file growth. If map file growth is a problem for you, try out this option instead.
We're doing more at Netflix with node.js analysis. Stay tuned, and also see the Netflix Tech Blog.
Click here for Disqus comments (ad supported).