Netflix
Performance Meetup

Global Client Performance
Fast Metrics

3G in Kazakhstan

Making the Internet fast
is slow.
Global Internet:
faster (better networking)
slower (broader reach, congestion)
Don't wait for it, measure it and deal
Working app >gt; Feature rich app

We need to know what the Internet looks like,
without averages, seeing the full distribution.

Logging Anti-Patterns
Averages
Sampling
Can't see the distribution
Missed data
Outliers heavily distort
Rare events
∞, 0, negatives, errors
Problems aren’t equal in
Population
Instead, use the client as a map-reducer and send up aggregated
data, less often.

Sizing up the Internet.

Infinite (free) compute power!

Get median, 95th, etc.
Calculate the inverse empirical cumulative
distribution function by math.
o ...or just use R which is free and knows how
to do it already
>gt; library(HistogramTools)
>gt; iecdf 

Data >gt; Opinions.

Better than debating opinions.
"We live in a
"No one really minds the
50ms world!"
spinner."
"Why should we spend
time on that instead of
COOLFEATURE?"
"There's no way that the
client makes that many
requests.”
Architecture is hard. Make it cheap to experiment where your users really are.

We built Daedalus
Fast
DNS Time
Elsewhere
Slow

Interpret the data
Visual → Numerical, need the IECDF for
Percentiles
ƒ(0.50) = 50th (median)
ƒ(0.95) = 95th
Cluster to get pretty colors similar experiences.
(k-means, hierarchical, etc.)

Practical Teleportation.
Go there!
Abstract analysis - hard
Feeling reality is much simpler than looking at graphs. Build!

Make a Reality Lab.

Don't guess.
Developing a model based on
production data, without missing the
distribution of samples (network, render,
responsiveness) will lead to better
software.
Global reach doesn't need to be scary.
@gcirino42 http://blogofsomeguy.com

Icarus
Martin Spier
@spiermar
Performance Engineering @ Netflix

Problem & Motivation
Real-user performance monitoring solution
More insight into the App performance
(as perceived by real users)
Too many variables to trust synthetic
tests and labs
Prioritize work around App performance
Track App improvement progress over time
Detect issues, internal and external

Device Diversity
● Netflix runs on all sorts of devices
● Smart TVs, Gaming Consoles, Mobile Phones, Cable TV boxes, ...
● Consistently evaluate performance

What are we monitoring?
User Actions
(or things users do in the App)
App Startup
User Navigation
Playing a Title
Internal App metrics

What are we measuring?
● When does the timer start and stop?
● Time-to-Interactive (TTI)
○ Interactive, even if
some items were not fully
loaded and rendered
● Time-to-Render (TTR)
○ Everything above the fold
(visible without scrolling)
is rendered
● Play Delay
● Meaningful for what we are monitoring

High-dimensional Data
● Complex device categorization
● Geo regions, subregions, countries
● Highly granular network
classifications
● High volume of A/B tests
● Different facets of the same user action
○ Cold, suspended and backgrounded
App startups
○ Target view/page on App startup

Data Sketches
● Data structures that approximately
resemble a much larger data set
● Preserve essential features!
● Significantly smaller!
● Faster to operate on!

t-Digest
● t-Digest data structure
● Rank-based statistics
(such as quantiles)
● Parallel friendly
(can be merged!)
● Very fast!
● Really accurate!
https://github.com/tdunning/t-digest

+ t-Digest sketches

iOS Median Comparison, Break by Country

iOS Median Comparison, Break by Country + iPhone 6S Plus

CDFs by UI Version

Warm Startup Rate

A/B Cell Comparison

Anomaly Detection

Going Forward
● Resource utilization metrics
● Device profiling
○ Instrumenting client code
● Explore other visualizations
○ Frequency heat maps
● Connection between perceived
performance, acquisition and
retention
@spiermar

Netflix
Autoscaling for experts
Vadim

Savings!
● Mid-tier stateless services are ~2/3rd of the total
● Savings - 30% of mid-tier footprint (roughly 30K instances)
○ Higher savings if we break it down by region
○ Even higher savings on services that scale well

Why we autoscale - philosophical reasons

Why we autoscale - pragmatic reasons
** Hack-day project
Encoding
Precompute
Failover
Red/black pushes
Curing cancer**
And more...

Should you autoscale?
Benefits
● On-demand capacity: direct $$ savings
● RI capacity: re-purposing spare capacity
However, for each server group, beware of
● Uneven distribution of traffic
● Sticky traffic
● Bursty traffic
● Small ASG sizes (

Autoscaling impacts availability - true or false?
* If done correctly
Under-provisioning, however, can impact availability
● Autoscaling is not a problem
● The real problem is not knowing performance characteristics of the
service

AWS autoscaling mechanics
ASG scaling policy
CloudWatch alarm
Aggregated metric feed
Notification
Tunables
Metric
● Threshold
● # of eval periods
● Scaling amount
● Warmup time

What metric to scale on?
Resource
utilization
Throughput
Pros
Tracks a direct measure of work
Linear scaling
Predictable
Requires less adjustment over time
Cons
Thresholds tend to drift over time
Prone to changes in request mixture
Less predictable
More oscillation / jitter

Autoscaling on multiple metrics
Proceed with caution
● Harder to reason about scaling behavior
● Different metrics might contradict each
other, causing oscillation
Typical Netflix configuration:
● Scale-up policy on throughput
● Scale-down policy on throughput
● Emergency scale-up policy on CPU, aka
“the hammer rule”

Well-behaved autoscaling

Common mistakes - “no rush” scaling
Problem: scaling amounts too
small, cooldown too long
Effect: scaling lags behind the
traffic flow. Not enough
capacity at peak, capacity
wasted in trough
Remedy: increase scaling
amounts, migrate to step
policies

Common mistakes - twitchy scaling
Problem: Scale-up policy is
too aggressive
Effect: unnecessary
capacity churn
Remedy: reduce scale-up
amount, increase the # of
eval periods

Common mistakes - should I stay or should I go
Problem: -up and -down
thresholds are too close to each
other
Effect: constant capacity
oscillation
Remedy: move -up and -down
thresholds farther apart

AWS target tracking - your best bet!
Think of it as a step policy with auto-steps
You can also think of it as a thermostat
Accounts for the rate of change in monitored metric
Pick a metric, set the target value and warmup time - that’s it!
Step
Target-tracking

Netflix
PMCs on the Cloud
Brendan

90% CPU utilization:
Busy
Waiting
(“idle”)

90% CPU utilization:
Waiting
(“idle”)
Busy
Reality:
Busy
Waiting
(“stalled”)
Waiting
(“idle”)

# perf stat -a -- sleep 10
Performance counter stats for 'system wide':
7,562
1,157
109,734
gt;
gt;
gt;
gt;
gt;
gt;
task-clock (msec)
context-switches
cpu-migrations
page-faults
cycles
stalled-cycles-frontend
stalled-cycles-backend
instructions
branches
branch-misses
10.001715965 seconds time elapsed
8.000 CPUs utilized
0.095 K/sec
0.014 K/sec
0.001 M/sec
(100.00%)
(100.00%)
(100.00%)
Performance
Monitoring Counters
(PMCs) in most clouds

# perf stat -a -- sleep 10
Performance counter stats for 'system wide':
641320.173626 task-clock (msec)
1,047,222 context-switches
83,420 cpu-migrations
38,905 page-faults
655,419,788,755 cycles
gt; stalled-cycles-frontend
gt; stalled-cycles-backend
536,830,399,277 instructions
97,103,651,128 branches
1,230,478,597 branch-misses
64.122 CPUs utilized
0.002 M/sec
0.130 K/sec
0.061 K/sec
1.022 GHz
[100.00%]
[100.00%]
[100.00%]
0.82 insns per cycle
151.412 M/sec
1.27% of all branches
[75.02%]
[75.02%]
[74.99%]
[75.02%]
10.001622154 seconds time elapsed
AWS EC2 m4.16xl

Interpreting IPC & Actionable Items
IPC: Instructions Per Cycle (invert of CPI)
● IPC gt; 1.0: likely instruction bound
Reduce code execution, eliminate unnecessary work, cache operations,
improve algorithm order. Can analyze using CPU flame graphs.
Faster CPUs.

Intel Architectural PMCs
Event Name
Umask
Event S.
Example Event Mask Mnemonic
UnHalted Core Cycles
00H
3CH
CPU_CLK_UNHALTED.THREAD_P
Instruction Retired
00H
C0H
INST_RETIRED.ANY_P
UnHalted Reference Cycles
01H
3CH
CPU_CLK_THREAD_UNHALTED.REF_XCLK
LLC Reference
4FH
2EH
LONGEST_LAT_CACHE.REFERENCE
LLC Misses
41H
2EH
LONGEST_LAT_CACHE.MISS
Branch Instruction Retired
00H
C4H
BR_INST_RETIRED.ALL_BRANCHES
Branch Misses Retired
00H
C5H
BR_MISP_RETIRED.ALL_BRANCHES
Now available in AWS EC2 on full dedicated hosts (eg, m4.16xl, …)

# pmcarch 1
CYCLES
[...]
INSTRUCTIONS
IPC BR_RETIRED
0.71 11760496978
0.78 10665897008
0.82 9538082731
0.78 12672090735
0.67 10542795714
BR_MISPRED
BMR% LLCREF
1.48 1542464817
1.48 1361315177
1.44 1272163733
1.43 1685112288
1.37 1204703117
LLCMISS
LLC%
https://github.com/brendangregg/pmc-cloud-tools
tiptop Tasks: 96 total,
PID [ %CPU] %SYS
35.3 28.5
1319+
[root]
3 displayed
Mcycle
screen
Minstr
IPC
%MISS
%BMIS
0: default
%BUS COMMAND
0.0 java
0.0 nm-applet
0.0 dbus-daemo

Netflix
Performance Meetup

Netflix
Performance Meetup