ACMApplicative2016_SystemMethodology.pdf

ACM Applicative 2016: System Methodology

Video: https://youtu.be/eO94l0aGLCA?t=3m37s

System Methodology - Holistic Performance Analysis on Modern Systems

Description: "Traditional systems performance engineering makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. For modern systems, we can choose the metrics, and can choose ones we need to support new holistic performance analysis methodologies. These methodologies provide faster, more accurate, and more complete analysis, and can provide a starting point for unfamiliar systems.

Methodologies are especially helpful for modern applications and their workloads, which can pose extremely complex problems with no obvious starting point. There are also continuous deployment environments such as the Netflix cloud, where these problems must be solved in shorter time frames. Fortunately, with advances in system observability and tracers, we have virtually endless custom metrics to aid performance analysis. The problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.

System methodologies provide a starting point for analysis, as well as guidance for quickly moving through the metrics to root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, and more.

This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. The focus is on single systems (any operating system), including single cloud instances, and quickly locating performance issues or exonerating the system. Many methodologies will be discussed, along with recommendations for their implementation, which may be as documented checklists of tools, or custom dashboards of supporting metrics. In general, you will learn to think differently about your systems, and how to ask better questions."

	next prev 1/57
	next prev 2/57
	next prev 3/57
	next prev 4/57
	next prev 5/57
	next prev 6/57
	next prev 7/57
	next prev 8/57
	next prev 9/57
	next prev 10/57
	next prev 11/57
	next prev 12/57
	next prev 13/57
	next prev 14/57
	next prev 15/57
	next prev 16/57
	next prev 17/57
	next prev 18/57
	next prev 19/57
	next prev 20/57
	next prev 21/57
	next prev 22/57
	next prev 23/57
	next prev 24/57
	next prev 25/57
	next prev 26/57
	next prev 27/57
	next prev 28/57
	next prev 29/57
	next prev 30/57
	next prev 31/57
	next prev 32/57
	next prev 33/57
	next prev 34/57
	next prev 35/57
	next prev 36/57
	next prev 37/57
	next prev 38/57
	next prev 39/57
	next prev 40/57
	next prev 41/57
	next prev 42/57
	next prev 43/57
	next prev 44/57
	next prev 45/57
	next prev 46/57
	next prev 47/57
	next prev 48/57
	next prev 49/57
	next prev 50/57
	next prev 51/57
	next prev 52/57
	next prev 53/57
	next prev 54/57
	next prev 55/57
	next prev 56/57
	next prev 57/57

PDF: ACMApplicative2016_SystemMethodology.pdf

Keywords (from pdftotext):

slide 1:

ACM Applicative 2016
Jun,	
  2016	
  
System	
  Methodology	
  
Holis0c	
  Performance	
  Analysis	
  on	
  
Modern	
  Systems	
  
Brendan Gregg
Senior Performance Architect

slide 2:

slide 3:

Apollo LMGC
performance analysis
CORE	
  SET	
  
AREA	
  
VAC	
  SETS	
  
ERASABLE	
  
MEMORY	
  
FIXED	
  
MEMORY

slide 4:

slide 5:

Background

slide 6:

History	
  
• System Performance Analysis up to the '90s:
– Closed source UNIXes and applications
– Vendor-created metrics and performance tools
– Users interpret given metrics
• Problems
– Vendors may not provide the best metrics
– Often had to infer, rather than measure
– Given metrics, what do we do with them?
# ps alx
F S UID
3 S
1 S
1 S
[…]
PID
PPID CPU PRI NICE
0 30
0 30
ADDR
WCHAN TTY TIME CMD
4412 ? 186:14 swapper
46520 ?
0:00 /etc/init
46554 co 0:00 –sh

slide 7:

Today	
  
1. Open source
Operating systems: Linux, BSDs, illumos, etc.
Applications: source online (Github)
2. Custom metrics
Can patch the open source, or,
Use dynamic tracing (open source helps)
3. Methodologies
Start with the questions, then make metrics to answer them
Methodologies can pose the questions
Biggest problem with dynamic tracing has been what to do with it.
Methodologies guide your usage.

slide 8:

Crystal	
  Ball	
  Thinking

slide 9:

An#-‐Methodologies

slide 10:

Street	
  Light	
  An#-‐Method	
  
1. Pick observability tools that are
– Familiar
– Found on the Internet
– Found at random
2. Run tools
3. Look for obvious issues

slide 11:

Drunk	
  Man	
  An#-‐Method	
  
• Drink Tune things at random until the problem goes away

slide 12:

Blame	
  Someone	
  Else	
  An#-‐Method	
  
1. Find a system or environment component you are not
responsible for
2. Hypothesize that the issue is with that component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1

slide 13:

Traﬃc	
  Light	
  An#-‐Method	
  
1. Turn all metrics into traffic lights
2. Open dashboard
3. Everything green? No worries, mate.
• Type I errors: red instead of green
– team wastes time
• Type II errors: green instead of red
– performance issues undiagnosed
– team wastes more time looking elsewhere
Traffic lights are suitable for objective metrics (eg, errors),
not subjective metrics (eg, IOPS, latency).

slide 14:

Methodologies

slide 15:

Performance	
  Methodologies	
  
• For system engineers:
– ways to analyze unfamiliar
systems and applications
• For app developers:
– guidance for metric and
dashboard design
Collect your
own toolbox of
methodologies
System Methodologies:
– Problem statement method
– Functional diagram method
– Workload analysis
– Workload characterization
– Resource analysis
– USE method
– Thread State Analysis
– On-CPU analysis
– CPU flame graph analysis
– Off-CPU analysis
– Latency correlations
– Checklists
– Static performance tuning
– Tools-based methods

slide 16:

Problem	
  Statement	
  Method	
  
1. What makes you think there is a performance problem?
2. Has this system ever performed well?
3. What has changed recently?
software? hardware? load?
4. Can the problem be described in terms of latency?
or run time. not IOPS or throughput.
5. Does the problem affect other people or applications?
6. What is the environment?
software, hardware, instance types?
versions? config?

slide 17:

Func0onal	
  Diagram	
  Method	
  
1. Draw the functional diagram
2. Trace all components in the data path
3. For each component, check performance
Breaks up a bigger problem into
smaller, relevant parts
Eg, imagine throughput
between the UCSB 360 and
the UTAH PDP10 was slow…
ARPA	
  Network	
  1969

slide 18:

Workload	
  Analysis	
  
• Begin with application metrics & context
• A drill-down methodology
Workload	
  
• Pros:
– Proportional,
accurate metrics
– App context
Applica0on	
  
	
  
	
   Libraries	
  
System	
  
• Cons:
– App specific
– Difficult to dig from
app to resource
System	
  Calls	
  
Kernel	
  
Hardware	
  
Analysis

slide 19:

Workload	
  Characteriza0on	
  
• Check the workload: who, why, what, how
– not resulting performance
Workload	
  
Target	
  
• Eg, for CPUs:
Who: which PIDs, programs, users
Why: code paths, context
What: CPU instructions, cycles
How: changing over time

slide 20:

Workload	
  Characteriza0on:	
  CPUs	
  
Who
Why
top
CPU	
  sample	
  
ﬂame	
  graphs	
  
How
What
monitoring	
  
PMCs

slide 21:

Resource	
  Analysis	
  
• Typical approach for system performance analysis:
begin with system tools & metrics
Workload	
  
• Pros:
– Generic
– Aids resource
perf tuning
Applica0on	
  
	
  
	
   Libraries	
  
System	
  
• Cons:
– Uneven coverage
– False positives
System	
  Calls	
  
Kernel	
  
Hardware	
  
Analysis

slide 22:

The	
  USE	
  Method	
  
• For every resource, check:
Utilization: busy time
Saturation: queue length or time
Errors: easy to interpret (objective)
Starts with the questions, then finds the tools
Eg, for hardware, check every resource incl. busses:

slide 23:

http://www.brendangregg.com/USEmethod/use-rosetta.html

slide 24:

slide 25:

Apollo Guidance
Computer
CORE	
  SET	
  
AREA	
  
VAC	
  SETS	
  
ERASABLE	
  
MEMORY	
  
FIXED	
  
MEMORY

slide 26:

USE	
  Method:	
  SoZware	
  
• USE method can also work for software resources
– kernel or app internals, cloud environments
– small scale (eg, locks) to large scale (apps). Eg:
• Mutex locks:
– utilization à lock hold time
– saturation à lock contention
– errors à any errors
X	
  
• Entire application:
– utilization à percentage of worker threads busy
– saturation à length of queued work
– errors à request errors
Resource	
  
U0liza0on	
  
(%)

slide 27:

RED	
  Method	
  
• For every service, check that:
Request rate
Error rate
Duration (distribution)
Metrics	
  
Database	
  
are within SLO/A
Another exercise in posing questions
from functional diagrams
Load	
  
Balancer	
  
Web	
  
Proxy	
  
Web	
  Server	
  
User	
  
Database	
  
Payments	
  
Server	
  
Asset	
  
Server	
  
By Tom Wilkie: http://www.slideshare.net/weaveworks/monitoring-microservices

slide 28:

Thread	
  State	
  Analysis	
  
State transition diagram
Identify & quantify
time in states
Narrows further
analysis to state
Thread states are
applicable to all apps

slide 29:

TSA:	
  eg,	
  Solaris

slide 30:

TSA:	
  eg,	
  RSTS/E	
  
RSTS: DEC OS
from the 1970's
TENEX (1969-72)
also had Control-T
for job states

slide 31:

TSA:	
  eg,	
  OS	
  X	
  
Instruments:	
  Thread	
  States

slide 32:

On-‐CPU	
  Analysis	
  
1. Split into user/kernel states
CPU	
  U0liza0on	
  
Heat	
  Map	
  
– /proc, vmstat(1)
2. Check CPU balance
– mpstat(1), CPU utilization heat map
3. Profile software
– User & kernel stack sampling (as a CPU flame graph)
4. Profile cycles, caches, busses
– PMCs, CPI flame graph

slide 33:

CPU	
  Flame	
  Graph	
  Analysis	
  
1. Take a CPU profile
2. Render it as a flame graph
3. Understand all software that is in >gt;1% of samples
Discovers issues by their CPU usage
- Directly: CPU consumers
- Indirectly: initialization
of I/O, locks, times, ...
Narrows target of study
to only running code
- See: "The Flame Graph",
CACM, June 2016
Flame	
  Graph

slide 34:

Java	
  Mixed-‐Mode	
  CPU	
  Flame	
  Graph	
  
• eg, Linux perf_events, with:
• Java –XX:+PreserveFramePointer
• Java perf-map-agent
Java	
  
GC	
  
JVM	
  
Kernel

slide 35:

CPI	
  Flame	
  Graph	
  
• Profile cycle stack traces and instructions or stalls separately
• Generate CPU flame graph (cycles) and color using other profile
• eg, FreeBSD: pmcstat
red	
  ==	
  instruc0ons	
  
blue	
  ==	
  stalls

slide 36:

Oﬀ-‐CPU	
  Analysis	
  
Analyze off-CPU time
via blocking code path:
Off-CPU flame graph
Often need wakeup
code paths as well…

slide 37:

Oﬀ-‐CPU	
  Time	
  Flame	
  Graph	
  
directory	
  read	
  
from	
  disk	
  
ﬁle	
  read	
  
from	
  disk	
  
fstat	
  from	
  disk	
  
path	
  read	
  from	
  disk	
  
pipe	
  write	
  
Trace blocking events with
kernel stacks & time blocked
(eg, using Linux BPF)
Oﬀ-‐CPU	
  0me	
  
Stack	
  depth

slide 38:

Wakeup	
  Time	
  Flame	
  Graph	
  
Who did the wakeup:
… can also associate wake-up stacks with off-CPU stacks
(eg, Linux 4.6: samples/bpf/offwaketime*)

slide 39:

Chain	
  Graphs	
  
Associate more than
one waker: the full
chain of wakeups
With enough stacks,
all paths lead to metal
An approach for
analyzing all off-CPU
issues

slide 40:

Latency	
  Correla0ons	
  
1. Measure latency
histograms at different
stack layers
2. Compare histograms
to find latency origin
Even better, use latency
heat maps
• Match outliers based on
both latency and time

slide 41:

Checklists:	
  eg,	
  Linux	
  Perf	
  Analysis	
  in	
  60s	
  
1. uptime
2. dmesg | tail
3. vmstat 1
4. mpstat -P ALL 1
5. pidstat 1
6. iostat -xz 1
7. free -m
8. sar -n DEV 1
9. sar -n TCP,ETCP 1
10. top
load	
  averages	
  
kernel	
  errors	
  
overall	
  stats	
  by	
  0me	
  
CPU	
  balance	
  
process	
  usage	
  
disk	
  I/O	
  
memory	
  usage	
  
network	
  I/O	
  
TCP	
  stats	
  
check	
  overview	
  
http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html

slide 42:

Checklists:	
  eg,	
  Neklix	
  perfvitals	
  Dashboard	
  
1.	
  RPS,	
  CPU	
  
2.	
  Volume	
  
3.	
  Instances	
  
4.	
  Scaling	
  
5.	
  CPU/RPS	
  
6.	
  Load	
  Avg	
  
7.	
  Java	
  Heap	
  
8.	
  ParNew	
  
9.	
  Latency	
  
10.	
  99th	
  0le

slide 43:

Sta0c	
  Performance	
  Tuning:	
  eg,	
  Linux

slide 44:

Tools-‐Based	
  Method	
  
1. Try all the tools! May be an anti-pattern. Eg, OS X:

slide 45:

Other	
  Methodologies	
  
Scientific method
5 Why's
Process of elimination
Intel's Top-Down Methodology
Method R

slide 46:

What	
  You	
  Can	
  Do

slide 47:

What	
  you	
  can	
  do	
  
1. Know what's now possible on modern systems
– Dynamic tracing: efficiently instrument any software
– CPU facilities: PMCs, MSRs (model specific registers)
– Visualizations: flame graphs, latency heat maps, …
2. Ask questions first: use methodologies to ask them
3. Then find/build the metrics
4. Build or buy dashboards to support methodologies

slide 48:

Dynamic	
  Tracing:	
  Eﬃcient	
  Metrics	
  
Eg, tracing TCP retransmits
Kernel	
  
Old way: packet capture
tcpdump	
   1.	
  read	
  
2.	
  dump	
  
buﬀer	
  
Analyzer	
   1.	
  read	
  
2.	
  process	
  
3.	
  print	
  
ﬁle	
  system	
  
send	
  
receive	
  
disks	
  
New way: dynamic tracing
Tracer	
  
1.	
  conﬁgure	
  
2.	
  read	
  
tcp_retransmit_skb()

slide 49:

Dynamic	
  Tracing:	
  Measure	
  Anything	
  
Those are Solaris/DTrace tools. Now becoming possible on all OSes:
FreeBSD & OS X DTrace, Linux BPF, Windows ETW

slide 50:

Performance	
  Monitoring	
  Counters	
  
Eg, FreeBSD PMC groups for Intel Sandy Bridge:

slide 51:

Visualiza0ons	
  
Eg, Disk I/O latency as a heat map, quantized in kernel:

slide 52:

USE	
  Method:	
  eg,	
  Neklix	
  Vector	
  
CPU:	
  
u0liza0on	
  
Network:	
  
u0liza0on	
  
Memory:	
  
u0liza0on	
  
Disk:	
  
load	
  
satura0on	
  
satura0on	
  
load	
  
satura0on	
  
u0liza0on	
  
satura0on

slide 53:

USE	
  Method:	
  To	
  Do	
  
Showing what is and is not commonly measured
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E	
  
U	
   S	
   E

slide 54:

CPU	
  Workload	
  Characteriza0on:	
  To	
  Do	
  
Showing what is and is not commonly measured
Who
Why
top,	
  htop
perf record -g
ﬂame	
  Graphs	
  
How
What
monitoring	
  
perf stat -a -d

slide 55:

Summary	
  
• It is the crystal ball age of performance observability
• What matters is the questions you want answered
• Methodologies are a great way to pose questions

slide 56:

References	
  &	
  Resources	
  
USE Method
– http://queue.acm.org/detail.cfm?id=2413037
– http://www.brendangregg.com/usemethod.html
TSA Method
– http://www.brendangregg.com/tsamethod.html
Off-CPU Analysis
– http://www.brendangregg.com/offcpuanalysis.html
– http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html
– http://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html
Static Performance Tuning, Richard Elling, Sun blueprint, May 2000
RED Method: http://www.slideshare.net/weaveworks/monitoring-microservices
Other system methodologies
– Systems Performance: Enterprise and the Cloud, Prentice Hall 2013
– http://www.brendangregg.com/methodology.html
– The Art of Computer Systems Performance Analysis, Jain, R., 1991
Flame Graphs
– http://queue.acm.org/detail.cfm?id=2927301
– http://www.brendangregg.com/flamegraphs.html
– http://techblog.netflix.com/2015/07/java-in-flames.html
Latency Heat Maps
– http://queue.acm.org/detail.cfm?id=1809426
– http://www.brendangregg.com/HeatMaps/latency.html
ARPA Network: http://www.computerhistory.org/internethistory/1960s
RSTS/E System User's Guide, 1985, page 4-5
DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD, Prentice Hall 2011
Apollo: http://www.hq.nasa.gov/office/pao/History/alsj/a11 http://www.hq.nasa.gov/alsj/alsj-LMdocs.html

slide 57:

ACM Applicative 2016
Questions?
http://slideshare.net/brendangregg
http://www.brendangregg.com
bgregg@netflix.com
@brendangregg
Feb	
  
Jun,	
  
2016	
  
2016