Performance Analysis Methodology

A performance analysis methodology is a procedure that you can follow to analyze system or application performance. These generally provide a starting point and then guidance to root cause, or causes. Different methodologies are suited for solving different classes of issues, and you may try more than one before accomplishing your goal. Analysis without a methodology can become a fishing expedition, where metrics are examined ad hoc, until the issue is found – if it is at all.

Methodologies can help with another difficult issue: when to give up. At any complex environment, spending too much time analyzing one performance problem takes time away from other problems and other possible wins. Performance engineers can spend six months working a single issue without success, with no end in sight, when it would be better to give up after a month and turn their attention elsewhere. So when should you give up? Up to you and the importance of the issue, but generally after I've tried several different methodologies I start to think I've given it a fair try and can start to look elsewhere. Which methodologies? Here's a suggested starting point: 1) Problem Statement Method, 2) Workload Characterization Method, 3) USE Method, 4) CPU Profile Method, 5) Off-CPU Analysis, 6) Time Division Method.

Methodologies documented in more detail on this site (beyond this summary page) are:

The following summarizes methodologies I've either created or encountered. You can print these all out as a cheetsheet/reminder.

Summaries

I first summarized and named various performance methodologies (mostly developed by me) for my USENIX LISA 2012 talk: Performance Analysis Methodology (PDF, slideshare, youtube, USENIX), then later documented them in my Systems Performance book, and the ACMQ article Thinking Methodically about Performance, which was also published in Communications of the ACM, Feb 2013. More detailed references are at the end of this page.

The following is my most up to date summary list, with methodologies enumerated. These begin with anti-methods, which are included for comparison, and not to follow.

Anti-Methodologies

Blame-Someone-Else Anti-Method

Streetlight Anti-Method

Drunk Man Anti-Method

Random Change Anti-Method

Passive Benchmarking Anti-Method

Traffic Light Anti-Method

Methodologies

Ad Hoc Checklist Method

Problem Statement Method

RTFM Method

Scientific Method

OODA Loop

Workload Characterization Method

Drill-Down Analysis Method

Process of Elimination

Time Division Method

5 Whys Performance Method

By-Layer Method

Tools Method

USE Method

RED Method

CPU Profile Method

Off-CPU Analysis

Stack Profile Method

TSA Method

Active Benchmarking Method

Method R

Performance Evaluation Steps

Capacity Planning Process

Intel Hierarchical Top-Down Performance Characterization Methodology

Performance Mantras

Benchmarking Checklist

References


Last Updated: 23-Jan-2025