LPC2019_BPF_Tracing

LPC2019 BPF Tracing Tools

Slides for the Linux Plumber's 2019 discussion on BPF tracing tools and future work.

	next prev 1/9
	next prev 2/9
	next prev 3/9
	next prev 4/9
	next prev 5/9
	next prev 6/9
	next prev 7/9
	next prev 8/9
	next prev 9/9

PDF: LPC2019_BPF_Tracing_Tools.pdf

Keywords (from pdftotext):

slide 1:

BPF Tracing Tools
Brendan Gregg

slide 2:

(brutal)
Ease of use (less brutal)
The Tracing Landscape, Sep 2019
(my opinion)
(eBPF)
(0.9.2)
bpftrace
ply/BPF
sysdig
(many)
perf
stap
LTTng
recent changes
(alpha)
(mature)
Stage of
Development
(hist
ftrace
synt riggers,
hetic
even
ts )
bcc/BPF
C/BPF
Raw BPF
Scope & Capability

slide 3:

BPF
Perf
Tools

slide 4:

BPF Perf Tools Example: readahead
Is readahead polluting the cache?
# readahead.bt
Attaching 5 probes...
Readahead unused pages: 128
Readahead used page age (ms):
@age_ms:
[1]
2455 |@@@@@@@@@@@@@@@
[2, 4)
8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)
4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@
[8, 16)
7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[16, 32)
4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@
[32, 64)
0 |
[64, 128)
0 |
[128, 256)
384 |@@

slide 5:

#!/usr/local/bin/bpftrace
kprobe:__do_page_cache_readahead
{ @in_readahead[tid] = 1; }
kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
kretprobe:__page_cache_alloc
/@in_readahead[tid]/
@birth[retval] = nsecs;
@rapages++;
kprobe:mark_page_accessed
/@birth[arg0]/
@age_ms = hist((nsecs - @birth[arg0]) / 1000000);
delete(@birth[arg0]);
@rapages--;
END
printf("\nReadahead unused pages: %d\n", @rapages);
printf("\nReadahead used page age (ms):\n");
print(@age_ms); clear(@age_ms);
clear(@birth); clear(@in_readahead); clear(@rapages);

slide 6:

Discussion: Desired Tracepoints
VFS
socket send/recv, skb alloc
tcp send/recv, udp send/recv
IP ECN
genl, bql
block:block_rq_issue/… add request pointer for use as unique ID
locks
to pair with skb:consume_skb/kfree_sub

slide 7:

Discussion: Desired BPF Helpers
struct file to pathname (like path_lookupat())
FD to struct file / pathname / file type (DF_SOCK etc)
bpf_get_current_pcomm()
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, …)
other timestamps
more string functions

slide 8:

Discussion: Bigger Capabilities
BTF (already there, thanks Yonghong Song etc)
unprivileged BPF
probe multi-attach (Ftrace is faster (__fentry__))
faster uprobes (LTTng-style)
bpf_probe_read_user/kernel split

slide 9:

Discussion: Challenges
libc no frame pointer
LBR+FP stack walking (but no LBR on the cloud (mostly))
JIT function tracing
update: user-level ORC a solution
Broken off-cpu flame graph (no frame pointer)