Recent posts:
Blog index

Brendan Gregg's Blog


node.js Flame Graphs on Linux

17 Sep 2014

CPU flame graphs are a useful visualization application stack traces, allowing you to quickly identify and quantify what to tune to improve performance. For Node.js they have solved countless problems on systems which have DTrace for sampling stack traces. But what about Linux?

At Netflix we have node.js in production at scale, on Linux instances in AWS EC2, and we create flame graphs using Linux perf_events and v8's --perf-basic-prof option. In this quick blog post, I'll share how it works and how you can do it, and what needs to be fixed to improve it further.

1. The problem

Using perf_events to profile CPU usage on node.js 0.10.23:

It's interactive: mouse-over elements for details, and click the SVG to zoom. The CPU flame graphs page explains how to interpret these, and this was created using the instructions in the Linux perf section.

This flame graph is partially-useful, as I can see system and v8 library symbols. However, it is missing JavaScript symbols (the blank rectangles), since v8, like the JVM, compiles and places symbols just in time (JIT).

2. Linux perf_events JIT support

In 2009, Linux perf_events added JIT symbol support, so that symbols from language virtual machines like the JVM could be inspected. It works in the following amazingly simple way:

  1. Your JIT application must be modified to create a /tmp/ file, which is a simple text database containing symbol addresses (in hex), sizes, and symbol names.
  2. That's it.

perf already looks for the /tmp/ file, and if it finds it, it uses it for symbol translations. So only v8 needed to be modified.

3. v8 --perf-basic-prof support

In November 2013, v8 added perf_events support, enabled using the --perf-basic-prof option. This made it into node v0.11.13. It works like this:

# ~/node-v0.11.13-linux-x64/bin/node --perf-basic-prof hello.js &
[1] 31441
# ls -l /tmp/
-rw-r--r-- 1 root root 81920 Sep 17 20:41 /tmp/
# tail /tmp/
14cec4db98a0 f Stub:BinaryOpICWithAllocationSiteStub(ADD_CreateAllocationMementos:String*Generic->String)
14cec4db9920 f Stub:BinaryOpICWithAllocationSiteStub(ADD_CreateAllocationMementos:String*String->String)
14cec4db99a0 f Stub:BinaryOpICWithAllocationSiteStub(ADD_CreateAllocationMementos:String*Smi->String)
14cec4db9a20 22c LazyCompile:~nextTick node.js:389
14cec4db9cc0 156 Stub:KeyedLoadElementStub
14cec4db9e80 22 KeyedLoadIC:
14cec4db9f20 22 KeyedLoadIC:
14cec4db9fc0 56 Stub:DoubleToIStub
14cec4dba080 10c Stub:KeyedStoreElementStub

This text file is what perf_events reads.

4. node.js Flame Graphs

Now that we have node 0.11.13+ running with --perf-basic-prof, we can create a flame graph using:

$ sudo bash
# perf record -F 99 -p `pgrep -n node` -g -- sleep 30
# perf script > out.nodestacks01
# git clone --depth 1
# cd FlameGraph
# ./ < ../out.nodestacks01 | ./ > ../out.nodestacks01.svg

You can also use stackvis, by Dave Pacheco, a node.js implementation which has extra features.

Here's an example result:

Note the JavaScript symbols are now readable. Click the SVG to zoom in. This actual flame graph isn't very interesting, as I'm just testing a dummy app to test out --perf-basic-prof.

Thanks to Trevor Norris for first posting the instructions for doing this in a short gist, which you may find useful to read. He also provides a script to facilitate this.

WARNING: map file growth

We can currently only use --perf-basic-prof for short periods (hours), due to bug 3453: the file can grow endlessly, eating Gbytes in a few days. It looks like symbols are moving location (they are supposed to stay put with --perf-basic-prof), causing the map file to keep growing.

So we can create flame graphs on Linux currently, but it will be ad hoc until that bug is fixed, and we can run with this option all the time.

If this bug is a nuisance to you, too, and it isn't yet fixed, please upvote that bug! If it's too painful to wait for the fix, you could run on an OS with DTrace, where node.js stack profiling doesn't have this issue (or, at least, run a canary instance for performance analysis).


We're doing more at Netflix with node.js analysis. Stay tuned, and also see the Netflix Tech Blog.

comments powered by Disqus