sar, benchmarking, tuning:
Images license: creative commons Attribution-ShareAlike 4.0.
This page links to various Linux performance material I've created, including the tools maps on the right. These show: Linux observability tools, Linux benchmarking tools, Linux tuning tools, and Linux sar. For more diagrams, see my slide decks below.
- perf_events: perf one-liners, examples, visualizations.
- perf-tools: perf analysis tools using perf_events and ftrace (github).
- ktap: one-liners, examples, and scripts.
- Flame Graphs: using perf, SystemTap, and ktap.
- My post Performance Tuning Linux Instances on EC2 includes the tunables we're using at Netflix (2015).
- My lwn.net article Ftrace: The Hidden Light Switch shows a use case for Linux ftrace (Aug, 2014).
- Posts about ftrace-based perf-tools: iosnoop for Linux, iosnoop Latency Heat Maps, opensnoop for Linux, execsnoop for Linux, tcpretrans, Page Cache Hit Ratio (2014).
- Posts about perf-based perf-tools: perf Hacktogram.
- Posts about perf_events: perf CPU Sampling, perf Static Tracepoints, perf Heat Maps, perf Counting, perf Kernel Line Tracing, perf off-CPU Time Flame Graphs, Linux Profiling at Netflix (2014-5).
- The blog post strace Wow Much Syscall discusses strace(1) for production use, and compares it to advanced tracing tools (2014).
- USE Method: Linux Performance Checklist; also see the USE Method page for the description of this methodology.
- Systems Performance: Enterprise and the Cloud (Prentice Hall, 2013) uses Linux distributions as the primary example.
LinuxCon Europe 2014
At LinuxCon Europe I gave a talk on Linux performance tools, summarizing the tool landscape in 50 minutes. My aim was to provide exposure to what exists: it isn't necessary to remember how (you can look that up later), but it is necessary to know that you can.
This was similar to my SCaLE11x talk, but this time I included benchmarking, tuning, and static tools for a more complete summary. I also updated my summary of the tracing tools, my recent work with ftrace, and included rdmsr, pcstat, and static tools. See my post for a longer summary.
At the Southern California Linux Expo (SCALE 13x), I gave a talk on Linux Profiling at Netflix using perf_events (aka "perf"), covering CPU profiling and a tour of other features. This talk covered gotchas, such as fixing stack traces and symbols when profiling Java and Node.js.
In a post about this talk, I included the interactive CPU flame graph SVG I was demonstrating.
At USENIX LISA 2014, I gave a talk on the new ftrace and perf_events tools I've been developing: the perf-tools collection on github.
The perf-tools collection, inspired by my earlier DTraceToolkit, provides advanced system performance analysis tools for Linux. Each tool has a man page and example file. They are unstable and unsupported, and they currently use shell scripting, hacks, and the ftrace and perf_events kernel tracing frameworks. They should be developed (and improved) as the Linux kernel acquires more tracing capabilities (eg, eBPF).
In a post about this talk, I included some more screenshots of these tools in action.
At SCaLE 12x (Southern California Linux Expo, 2014) I gave the keynote on What Linux can learn from Solaris performance and vice-versa. This drew on my experience analyzing Linux vs SmartOS performance, and my work for the Systems Performance book. SmartOS is an OS based on the illumos kernel, which is the active fork of the OpenSolaris kernel.
I've never seen a good talk comparing performance features of both, I suspect in part because it's hard to know them both in enough depth, and also hard to choose from the many differences which should be highlighted.
This presentation also contains ponies. Lots of ponies. These are the unofficial mascots for DTrace, perf_events, SystemTap, ktap, and LTTng, and were designed by the same person (Deirdré) who designed the original (and popular) DTrace ponycorn.
At SCaLE 11x (2013) I gave a talk on Linux performance analysis and tools, and described over twenty observability tools, and then methodologies for using them. During this talk I introduced the Linux performance tools diagram, which I've put at the top of this page (an updated version).
I've used pretty much everything for solving performance issues, including advanced tools like perf, DTrace, and SystemTap, and I explained their role and how they fit together. It was pretty dense: you can treat this as a 60 minute crash course into Linux performance analysis and tools.
Thanks Deirdré Straughan for filming it and then spreading the word afterwards.
Other resources (not by me) I'd recommend for the topic of Linux performance:
- Linux Instrumentation: slides from a great talk in June 2010 by Ian Munsie, which summarizes the different Linux tracers very well. If you're trying to understand all the tracers and frameworks, this is worth studying (keeping in mind it's from 2010).