ACM Applicative 2016: System Methodology
Video: https://youtu.be/eO94l0aGLCA?t=3m37sSystem Methodology - Holistic Performance Analysis on Modern Systems
Description: "Traditional systems performance engineering makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. For modern systems, we can choose the metrics, and can choose ones we need to support new holistic performance analysis methodologies. These methodologies provide faster, more accurate, and more complete analysis, and can provide a starting point for unfamiliar systems.
Methodologies are especially helpful for modern applications and their workloads, which can pose extremely complex problems with no obvious starting point. There are also continuous deployment environments such as the Netflix cloud, where these problems must be solved in shorter time frames. Fortunately, with advances in system observability and tracers, we have virtually endless custom metrics to aid performance analysis. The problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.
System methodologies provide a starting point for analysis, as well as guidance for quickly moving through the metrics to root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, and more.
This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. The focus is on single systems (any operating system), including single cloud instances, and quickly locating performance issues or exonerating the system. Many methodologies will be discussed, along with recommendations for their implementation, which may be as documented checklists of tools, or custom dashboards of supporting metrics. In general, you will learn to think differently about your systems, and how to ask better questions."
next prev 1/57 | |
next prev 2/57 | |
next prev 3/57 | |
next prev 4/57 | |
next prev 5/57 | |
next prev 6/57 | |
next prev 7/57 | |
next prev 8/57 | |
next prev 9/57 | |
next prev 10/57 | |
next prev 11/57 | |
next prev 12/57 | |
next prev 13/57 | |
next prev 14/57 | |
next prev 15/57 | |
next prev 16/57 | |
next prev 17/57 | |
next prev 18/57 | |
next prev 19/57 | |
next prev 20/57 | |
next prev 21/57 | |
next prev 22/57 | |
next prev 23/57 | |
next prev 24/57 | |
next prev 25/57 | |
next prev 26/57 | |
next prev 27/57 | |
next prev 28/57 | |
next prev 29/57 | |
next prev 30/57 | |
next prev 31/57 | |
next prev 32/57 | |
next prev 33/57 | |
next prev 34/57 | |
next prev 35/57 | |
next prev 36/57 | |
next prev 37/57 | |
next prev 38/57 | |
next prev 39/57 | |
next prev 40/57 | |
next prev 41/57 | |
next prev 42/57 | |
next prev 43/57 | |
next prev 44/57 | |
next prev 45/57 | |
next prev 46/57 | |
next prev 47/57 | |
next prev 48/57 | |
next prev 49/57 | |
next prev 50/57 | |
next prev 51/57 | |
next prev 52/57 | |
next prev 53/57 | |
next prev 54/57 | |
next prev 55/57 | |
next prev 56/57 | |
next prev 57/57 |
PDF: ACMApplicative2016_SystemMethodology.pdf
Keywords (from pdftotext):
slide 1:
ACM Applicative 2016 Jun, 2016 System Methodology Holis0c Performance Analysis on Modern Systems Brendan Gregg Senior Performance Architectslide 2:
slide 3:
Apollo LMGC performance analysis CORE SET AREA VAC SETS ERASABLE MEMORY FIXED MEMORYslide 4:
slide 5:
Backgroundslide 6:
History • System Performance Analysis up to the '90s: – Closed source UNIXes and applications – Vendor-created metrics and performance tools – Users interpret given metrics • Problems – Vendors may not provide the best metrics – Often had to infer, rather than measure – Given metrics, what do we do with them? # ps alx F S UID 3 S 1 S 1 S […] PID PPID CPU PRI NICE 0 30 0 30 ADDR WCHAN TTY TIME CMD 4412 ? 186:14 swapper 46520 ? 0:00 /etc/init 46554 co 0:00 –shslide 7:
Today 1. Open source Operating systems: Linux, BSDs, illumos, etc. Applications: source online (Github) 2. Custom metrics Can patch the open source, or, Use dynamic tracing (open source helps) 3. Methodologies Start with the questions, then make metrics to answer them Methodologies can pose the questions Biggest problem with dynamic tracing has been what to do with it. Methodologies guide your usage.slide 8:
Crystal Ball Thinkingslide 9:
An#-‐Methodologiesslide 10:
Street Light An#-‐Method 1. Pick observability tools that are – Familiar – Found on the Internet – Found at random 2. Run tools 3. Look for obvious issuesslide 11:
Drunk Man An#-‐Method • Drink Tune things at random until the problem goes awayslide 12:
Blame Someone Else An#-‐Method 1. Find a system or environment component you are not responsible for 2. Hypothesize that the issue is with that component 3. Redirect the issue to the responsible team 4. When proven wrong, go to 1slide 13:
Traffic Light An#-‐Method 1. Turn all metrics into traffic lights 2. Open dashboard 3. Everything green? No worries, mate. • Type I errors: red instead of green – team wastes time • Type II errors: green instead of red – performance issues undiagnosed – team wastes more time looking elsewhere Traffic lights are suitable for objective metrics (eg, errors), not subjective metrics (eg, IOPS, latency).slide 14:
Methodologiesslide 15:
Performance Methodologies • For system engineers: – ways to analyze unfamiliar systems and applications • For app developers: – guidance for metric and dashboard design Collect your own toolbox of methodologies System Methodologies: – Problem statement method – Functional diagram method – Workload analysis – Workload characterization – Resource analysis – USE method – Thread State Analysis – On-CPU analysis – CPU flame graph analysis – Off-CPU analysis – Latency correlations – Checklists – Static performance tuning – Tools-based methodsslide 16:
Problem Statement Method 1. What makes you think there is a performance problem? 2. Has this system ever performed well? 3. What has changed recently? software? hardware? load? 4. Can the problem be described in terms of latency? or run time. not IOPS or throughput. 5. Does the problem affect other people or applications? 6. What is the environment? software, hardware, instance types? versions? config?slide 17:
Func0onal Diagram Method 1. Draw the functional diagram 2. Trace all components in the data path 3. For each component, check performance Breaks up a bigger problem into smaller, relevant parts Eg, imagine throughput between the UCSB 360 and the UTAH PDP10 was slow… ARPA Network 1969slide 18:
Workload Analysis • Begin with application metrics & context • A drill-down methodology Workload • Pros: – Proportional, accurate metrics – App context Applica0on Libraries System • Cons: – App specific – Difficult to dig from app to resource System Calls Kernel Hardware Analysisslide 19:
Workload Characteriza0on • Check the workload: who, why, what, how – not resulting performance Workload Target • Eg, for CPUs: Who: which PIDs, programs, users Why: code paths, context What: CPU instructions, cycles How: changing over timeslide 20:
Workload Characteriza0on: CPUs Who Why top CPU sample flame graphs How What monitoring PMCsslide 21:
Resource Analysis • Typical approach for system performance analysis: begin with system tools & metrics Workload • Pros: – Generic – Aids resource perf tuning Applica0on Libraries System • Cons: – Uneven coverage – False positives System Calls Kernel Hardware Analysisslide 22:
The USE Method • For every resource, check: Utilization: busy time Saturation: queue length or time Errors: easy to interpret (objective) Starts with the questions, then finds the tools Eg, for hardware, check every resource incl. busses:slide 23:
http://www.brendangregg.com/USEmethod/use-rosetta.htmlslide 24:
slide 25:
Apollo Guidance Computer CORE SET AREA VAC SETS ERASABLE MEMORY FIXED MEMORYslide 26:
USE Method: SoZware • USE method can also work for software resources – kernel or app internals, cloud environments – small scale (eg, locks) to large scale (apps). Eg: • Mutex locks: – utilization à lock hold time – saturation à lock contention – errors à any errors X • Entire application: – utilization à percentage of worker threads busy – saturation à length of queued work – errors à request errors Resource U0liza0on (%)slide 27:
RED Method • For every service, check that: Request rate Error rate Duration (distribution) Metrics Database are within SLO/A Another exercise in posing questions from functional diagrams Load Balancer Web Proxy Web Server User Database Payments Server Asset Server By Tom Wilkie: http://www.slideshare.net/weaveworks/monitoring-microservicesslide 28:
Thread State Analysis State transition diagram Identify & quantify time in states Narrows further analysis to state Thread states are applicable to all appsslide 29:
TSA: eg, Solarisslide 30:
TSA: eg, RSTS/E RSTS: DEC OS from the 1970's TENEX (1969-72) also had Control-T for job statesslide 31:
TSA: eg, OS X Instruments: Thread Statesslide 32:
On-‐CPU Analysis 1. Split into user/kernel states CPU U0liza0on Heat Map – /proc, vmstat(1) 2. Check CPU balance – mpstat(1), CPU utilization heat map 3. Profile software – User & kernel stack sampling (as a CPU flame graph) 4. Profile cycles, caches, busses – PMCs, CPI flame graphslide 33:
CPU Flame Graph Analysis 1. Take a CPU profile 2. Render it as a flame graph 3. Understand all software that is in >gt;1% of samples Discovers issues by their CPU usage - Directly: CPU consumers - Indirectly: initialization of I/O, locks, times, ... Narrows target of study to only running code - See: "The Flame Graph", CACM, June 2016 Flame Graphslide 34:
Java Mixed-‐Mode CPU Flame Graph • eg, Linux perf_events, with: • Java –XX:+PreserveFramePointer • Java perf-map-agent Java GC JVM Kernelslide 35:
CPI Flame Graph • Profile cycle stack traces and instructions or stalls separately • Generate CPU flame graph (cycles) and color using other profile • eg, FreeBSD: pmcstat red == instruc0ons blue == stallsslide 36:
Off-‐CPU Analysis Analyze off-CPU time via blocking code path: Off-CPU flame graph Often need wakeup code paths as well…slide 37:
Off-‐CPU Time Flame Graph directory read from disk file read from disk fstat from disk path read from disk pipe write Trace blocking events with kernel stacks & time blocked (eg, using Linux BPF) Off-‐CPU 0me Stack depthslide 38:
Wakeup Time Flame Graph Who did the wakeup: … can also associate wake-up stacks with off-CPU stacks (eg, Linux 4.6: samples/bpf/offwaketime*)slide 39:
Chain Graphs Associate more than one waker: the full chain of wakeups With enough stacks, all paths lead to metal An approach for analyzing all off-CPU issuesslide 40:
Latency Correla0ons 1. Measure latency histograms at different stack layers 2. Compare histograms to find latency origin Even better, use latency heat maps • Match outliers based on both latency and timeslide 41:
Checklists: eg, Linux Perf Analysis in 60s 1. uptime 2. dmesg | tail 3. vmstat 1 4. mpstat -P ALL 1 5. pidstat 1 6. iostat -xz 1 7. free -m 8. sar -n DEV 1 9. sar -n TCP,ETCP 1 10. top load averages kernel errors overall stats by 0me CPU balance process usage disk I/O memory usage network I/O TCP stats check overview http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.htmlslide 42:
Checklists: eg, Neklix perfvitals Dashboard 1. RPS, CPU 2. Volume 3. Instances 4. Scaling 5. CPU/RPS 6. Load Avg 7. Java Heap 8. ParNew 9. Latency 10. 99th 0leslide 43:
Sta0c Performance Tuning: eg, Linuxslide 44:
Tools-‐Based Method 1. Try all the tools! May be an anti-pattern. Eg, OS X:slide 45:
Other Methodologies Scientific method 5 Why's Process of elimination Intel's Top-Down Methodology Method Rslide 46:
What You Can Doslide 47:
What you can do 1. Know what's now possible on modern systems – Dynamic tracing: efficiently instrument any software – CPU facilities: PMCs, MSRs (model specific registers) – Visualizations: flame graphs, latency heat maps, … 2. Ask questions first: use methodologies to ask them 3. Then find/build the metrics 4. Build or buy dashboards to support methodologiesslide 48:
Dynamic Tracing: Efficient Metrics Eg, tracing TCP retransmits Kernel Old way: packet capture tcpdump 1. read 2. dump buffer Analyzer 1. read 2. process 3. print file system send receive disks New way: dynamic tracing Tracer 1. configure 2. read tcp_retransmit_skb()slide 49:
Dynamic Tracing: Measure Anything Those are Solaris/DTrace tools. Now becoming possible on all OSes: FreeBSD & OS X DTrace, Linux BPF, Windows ETWslide 50:
Performance Monitoring Counters Eg, FreeBSD PMC groups for Intel Sandy Bridge:slide 51:
Visualiza0ons Eg, Disk I/O latency as a heat map, quantized in kernel:slide 52:
USE Method: eg, Neklix Vector CPU: u0liza0on Network: u0liza0on Memory: u0liza0on Disk: load satura0on satura0on load satura0on u0liza0on satura0onslide 53:
USE Method: To Do Showing what is and is not commonly measured U S E U S E U S E U S E U S E U S E U S E U S E U S E U S E U S Eslide 54:
CPU Workload Characteriza0on: To Do Showing what is and is not commonly measured Who Why top, htop perf record -g flame Graphs How What monitoring perf stat -a -dslide 55:
Summary • It is the crystal ball age of performance observability • What matters is the questions you want answered • Methodologies are a great way to pose questionsslide 56:
References & Resources USE Method – http://queue.acm.org/detail.cfm?id=2413037 – http://www.brendangregg.com/usemethod.html TSA Method – http://www.brendangregg.com/tsamethod.html Off-CPU Analysis – http://www.brendangregg.com/offcpuanalysis.html – http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html – http://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html Static Performance Tuning, Richard Elling, Sun blueprint, May 2000 RED Method: http://www.slideshare.net/weaveworks/monitoring-microservices Other system methodologies – Systems Performance: Enterprise and the Cloud, Prentice Hall 2013 – http://www.brendangregg.com/methodology.html – The Art of Computer Systems Performance Analysis, Jain, R., 1991 Flame Graphs – http://queue.acm.org/detail.cfm?id=2927301 – http://www.brendangregg.com/flamegraphs.html – http://techblog.netflix.com/2015/07/java-in-flames.html Latency Heat Maps – http://queue.acm.org/detail.cfm?id=1809426 – http://www.brendangregg.com/HeatMaps/latency.html ARPA Network: http://www.computerhistory.org/internethistory/1960s RSTS/E System User's Guide, 1985, page 4-5 DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD, Prentice Hall 2011 Apollo: http://www.hq.nasa.gov/office/pao/History/alsj/a11 http://www.hq.nasa.gov/alsj/alsj-LMdocs.htmlslide 57:
ACM Applicative 2016 Questions? http://slideshare.net/brendangregg http://www.brendangregg.com bgregg@netflix.com @brendangregg Feb Jun, 2016 2016