BSidesSF 2017: Security Monitoring with eBPF
Talk by Alex Maestretti and Brendan Gregg for BSidesSF.Video: https://www.youtube.com/watch?v=44nV6Mj11uw
Description: "Using the new extended Berkley Packet Filter capabilities in Linux to the improve performance of auditing security relevant kernel events around network, file and process actions."
This was the first talk to explore using eBPF for host-based intrusion detection. Slides 8 quotes from a phrack article by elfmaster.
 | next prev 1/27 |
 | next prev 2/27 |
 | next prev 3/27 |
 | next prev 4/27 |
 | next prev 5/27 |
 | next prev 6/27 |
 | next prev 7/27 |
 | next prev 8/27 |
 | next prev 9/27 |
 | next prev 10/27 |
 | next prev 11/27 |
 | next prev 12/27 |
 | next prev 13/27 |
 | next prev 14/27 |
 | next prev 15/27 |
 | next prev 16/27 |
 | next prev 17/27 |
 | next prev 18/27 |
 | next prev 19/27 |
 | next prev 20/27 |
 | next prev 21/27 |
 | next prev 22/27 |
 | next prev 23/27 |
 | next prev 24/27 |
 | next prev 25/27 |
 | next prev 26/27 |
 | next prev 27/27 |
PDF: BSidesSF2017_BPF_security_monitoring.pdf
Keywords (from pdftotext):
slide 1:
Security Monitoring with eBPF ALEX MAESTRETTI - MANAGER, SIRT BRENDAN GREGG - Sr ARCHITECT, PERFORMANCEslide 2:
The Brief. Extended Berkley Packet Filter (eBPF) is a new Linux feature which allows safe and efficient monitoring of kernel functions. This has dramatic implications for security monitoring, especially at Netflix scale. We are encouraging the security community to leverage this new technology to all of our benefit.slide 3:
Existing Solutions. There are many security monitoring solutions available today that meet a wide range of requirements. Our design goals were: push vs poll, lightweight, with kernel-level inspection. Our environment is composed of micro-services running on ephemeral and immutable instances built and deployed from source control into a public cloud. osquery ossec sysdig auditdslide 4:
A new Option.slide 5:
SCREENSHOT # capable TIME UID 22:11:23 114 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:23 0 22:11:24 0 22:11:24 0 22:11:24 0 [...] PID COMM snmpd run chmod chmod chmod chmod chown chown setuidgid setuidgid setuidgid run chmod chmod CAP NAME CAP_NET_ADMIN CAP_SYS_RESOURCE CAP_FOWNER CAP_FSETID CAP_FSETID CAP_FSETID CAP_FSETID CAP_FSETID CAP_SETGID CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE CAP_FOWNER CAP_FSETID AUDIT Snooping on Linux cap_capable() calls using bcc/eBPFslide 6:
SCREENSHOT # argdist -i 5 -C 'p::cap_capable():int:ctx->gt;dx' [06:32:08] p::cap_capable():int:ctx->gt;dx COUNT EVENT ctx->gt;dx = 35 ctx->gt;dx = 21 ctx->gt;dx = 12 [06:32:13] p::cap_capable():int:ctx->gt;dx COUNT EVENT ctx->gt;dx = 1 ctx->gt;dx = 21 ctx->gt;dx = 12 [...] Now frequency counting in-kernel and only sending the summary to user eBPF is much more than just a per-event tracer (this is a bcc/eBPF hack; I should make this into a real tool like the previous one)slide 7:
LINUX TRACING TIMELINE 2004: kprobes (2.6.9) 2005: DTrace (not Linux); SystemTap (out-of-tree) 2008: ftrace (2.6.27) 2009: perf_events (2.6.31) 2009: tracepoints (2.6.32) 2010-2016: ftrace & perf_events enhancements 2012: uprobes (3.5) 2014-2016: Enhanced BPF patches + other out of tree tracers LTTng, ktap, sysdig, ...slide 8:
KERNEL INSTRUMENTATION USING KPROBES PHRACK ZINE #67/6 2010-11-17 1 - Introduction 1.1 - Why write it? 1.2 - About kprobes "So why write this? Because... 1.3 - Jprobe example weprobe are patching hackers. Hackers should 1.4 - Kretprobe example & Return technique be aware of any and all 2 - Kprobes implementation 2.1 - Kprobe implementation resources available to them -2.2 - Jprobe implementation some more auspicious than 2.3 - File hiding with jprobes/kretprobes and modifying kernel .text 2.4 - Kretprobe implementation others -- Nonetheless, kprobes areread-only a sweet dealsegments when you 2.5 - A quick stop into modifying kernel 2.6 - An idea for a kretprobe implementation for hackers consider that they are a 3 - Patch to unpatch W^X (mprotect/mmap restrictions) native kernel API…" 4 - Notes on rootkit detection for kprobes 5 - Summing it all up. 6 - Greetz http://phrack.org/issues/67/6.html 7 - References and citations (also see http://phrack.org/issues/63/3.html) 8 - Codeslide 9:
BERKELEY PACKET FILTER # tcpdump host 127.0.0.1 and port 22 -d (000) ldh [12] (001) jeq #0x800 jt 2 jf 18 (002) ld [26] (003) jeq #0x7f000001 jt 6 jf 4 (004) ld [30] (005) jeq #0x7f000001 jt 6 jf 18 (006) ldb [23] (007) jeq #0x84 jt 10 jf 8 (008) jeq #0x6 jt 10 jf 9 (009) jeq #0x11 jt 10 jf 18 (010) ldh [20] (011) jset #0x1fff jt 18 jf 12 (012) ldxb 4*([14]&0xf) [...] 2 x 32-bit registers & scratch memory User-defined bytecode executed by an in-kernel sandboxed virtual machine Steven McCanne and Van Jacobson, 1993slide 10:
ENHANCED BPF (eBPF) 10 x 64-bit registers maps (hashes) actions Alexei Starovoitov, 2015+ There are front-ends (eg, bcc) so we never have to write such raw eBPFslide 11:
eBPF USE CASESslide 12:
BPF SECURITY MODULEslide 13:
WHAT TO MONITOR Trace low-frequency events wherever possible to lower overhead Eg, TCP connection init; not TCP send/receiveslide 14:
BCC EXAMPLES These bcc/BPF observability tools show what is possibleslide 15:
SCREENSHOT # ./execsnoop -x PCOMM PID supervise mkdir run chown run run [...] From the bcc collection RET ARGS 0 ./run 0 /bin/mkdir -p ./main 0 ./run 0 /bin/chown nobody:nobody ./main 0 /bin/mkdir -p ./main -2 /usr/local/bin/setuidgid nobody # ./tcpconnect -t TIME(s) PID COMM local_agent local_agent local_agent local_agent local_agent [...] IP SADDR 4 10.103.219.236 4 10.103.219.236 4 10.103.219.236 4 10.103.219.236 4 10.103.219.236 DADDR DPORTslide 16:
INSTRUMENTATION TECHNIQUES Use the stable-ist API possible In order of preference: Kernel events Tracepoints: stable API, if available. Kprobes: dynamic tracing of security hooks Kprobes: dynamic tracing of kernel functions User events User Statically Defined Tracing (USDT) probes: stable API, if available Uprobes: dynamic tracing of API interface functions Uprobes: dynamic tracing of internal functionsslide 17:
WHY eBPF ROCKS Safe Kernel verifies eBPF code (DAG and null reference check) Kernel memory access controlled through helper functions Part of the mainline kernel, no 3rd party kernel modules Flexible Add new instrumentation to production servers anytime Any event, any data Performant JIT’d instrumentation Data from kernel to user via async maps or per-events on a ring buffer Custom filters and summaries in kernel Preliminary results of logging TCP accept() to Can choose lower-frequency events to trace the file system, with a certain workload, and comparing overheads. Active benchmarking was performed. Each of these can likely be tuned further: results are not final.slide 18:
eBPF EFFICIENCY Eg, tracing TCP retransmits Old way: packet capture New way: dynamic tracingslide 19:
WRITING A bcc/eBPF PROGRAM What is in a bcc eBPF Python file: ● Python code for userland reporting ● eBPF C code for event handling, in a variable (or file) ● BCC calls to initialize BPF and probes bitehist.py example BPF Compiler Collection github.com/iovisor/bcc/slide 20:
ADVANCED eBPF It gets more complicated... from tcpaccept.pyslide 21:
Summary.slide 22:
MONITORING TO DETECTIONslide 23:
Thank you.slide 24:
Bonus round.slide 25:
WHAT’S YOUR SIGN (SYMBOL) Example: I want to detect unusual listening ports and what process has bound them. Let’s look at the socket lifecycle… ○ socket() is too early, no port yet ○ bind() and listen() are good candidates ○ if access is the only concern, accept() We can find kernel symbols a number of ways ○ List them: sudo cat /proc/kallsyms ○ Use perf-tools to trace ex. nc -l 12345 inet_ is the subsystem hooked in BCC examples and seems to have the context we need… but is not guaranteed stable across Linux builds. usna.eduslide 26:
PROTIP: HOOK THE LSM Most of the relevant functions we care about are already passing through the LSM (with good context), let’s Kprobe there (if we can’t find a tracepoint) as it will be more stable: /include/linux/security.hslide 27:
The end end.