I gave this demo recently at USENIX/LISA 2016, showing ftrace, perf, and bcc/BPF. A video is on youtube:
It was part of a larger talk on Linux 4.x Tracing Tools using BPF Superpowers. The slides are on slideshare:
The full talk video should be posted on the usenix website at some point.
My 15 (18) minute demo stepped through the evolution of recent built in Linux tracers: ftrace (2008+) and its many capabilities, perf (2009+), and bcc/BPF (2015+) which provides the final programmatic abilities for advanced tracing. I suspect I might change people's view of Linux tracing, as these tracers – despite being built in to the Linux kernel – are still not widely known.
Earlier at the conference, Sasha Goldshtein and I ran a half day perf & bcc/BPF tutorial. Both Sasha and I are not only bcc contributors, but also experienced classroom instructors, and it was a pleasure to collaborate with him on this project. It wasn't videoed, but the lab files are on github. If you are interested in learning bcc/BPF, there are also two tutorials I wrote in bcc/docs for using and developing bcc tools.
There was a lot of interest in both our tutorial and my talk – I imagine this interest will grow over time as more people deploy on Linux 4.x series kernels and can make use of BPF.
For more about Linux tracers, here are some resources (mostly my own):
- ftrace: The hidden light switch (lwn.net), perf-tools (github), ftrace.txt
- perf: perf Examples, perf wiki
- bcc: bcc/BPF tools, and many posts here (listed on Linux Performance).
Then there's also the add on tracers, like Systemtap, LTTng, sysdig, etc, which I didn't cover in 15 minutes.
My 15 minute tracing demo was inspired by Greg Law's excellent cppcon talk Give me 15 minutes & I'll change your view of GDB. Since then, I've also written about GDB here, with a full GDB example (tutorial).
LISA was a lot of fun. Thanks to those who were able to attend our events, and USENIX for putting on another great conference!