Jump to: Documentation Videos Software Misc

G'Day, I'm Brendan. I use this page to share links and downloads from my hobbies and work with computers. This work has included system administration, Unix security, Solaris, Linux, programming and kernel engineering. These days I work on performance and live in San Francisco.

I have two blogs: personal and work. I'm also on twitter and linkedin.

Documentation

Various things I've written, roughly from most to least recent, include:




• The introduction of a new visualization type: Subsecond Offset Heat Maps, which allow behavior within a second to be seen.
• The USE Method for identifying common system bottlenecks and errors, a methodology I developed and have used successfully for many years for enterprise and cloud performance analysis. Based on the USE method: the Solaris Performance Checklist and Linux Performance Checklist.
• The Flame Graph visualization and using them for Linux Kernel Performance.
• Performance visualizations: Device Utilization, and a series of three (so far) on Visualizing the Cloud, Process Snapshots and Process Execution.
• Narrow topics in operating system performance: Activity of the ZFS ARC.
• A long post about Using SystemTap on the Ubuntu and CentOS Linux distributions.
• An article to introduce the technique of Off-CPU Performance Analysis, which can identify the cause of high latency due to blocking events.
• Top 10 DTrace Scripts for Mac OS X performance analysis and troubleshooting, written to reach the broader Mac OS X community. This includes step by step instructions on how to find and run the Terminal application and sudo.
• A series of blog posts on File System latency, using MySQL as an example application (1, 2, 3, 4, 5).
• MySQL Query Latency using DTrace.
• A series of blog posts on the DTrace pid provider, going beyond what was covered in the DTrace book.
• The DTrace book with Jim Mauro, Prentice Hall 2011 (ISBN 0132091518). A sample chapter on File Systems is online. This 1152 page book took over a year to write, including the research, development and testing of dozens of new DTrace scripts and one-liners, and soliciting input from many experts.
• The Visualizations for Performance Analysis slide deck, for USENIX/LISA 2010 (video also available). This illustrates visuals for performance analysis, showing what is effective and becomming important for cloud computing environments.
• An article for ACMQ, also published by CACM, on Visualizing System latency. This includes interesting latency heat maps I had found, including the Rainbow Pterodactyl and the Icy Lake.
• A series of posts on performance testing a series of storage appliances (1, 2, 3). These were written at a time when I was often pulled in to save benchmarking mishaps, and needed to share tips to avoid common mistakes. They were very successful (and thanks to those who read them) as the calls for help were greatly reduced.
• The storage appliance Dashboard where I used weather icons to highlight performance issues and convey ambiguity for those types of metric.
• The original ZFS L2ARC post and later L2ARC Screenshots. Since code changes were public each night, my block comment in usr/src/uts/common/fs/zfs/arc.c (added in Nov 2007) disguised the then-secret intent of this technology by listing "short-stroked disks" as the first intended device, instead of SSDs.
• The companion to Solaris Internals 2nd Edition: Solaris Performance and Tools, with Richard McDougall and Jim Mauro, Prentice Hall 2006 (ISBN 0131568191).
• My Solaris 10 Zones page: the first showing how to configure Solaris Zones with Resource Controls (which I had figured out the hard way).
• A page on DTrace, where I described and shared early scripts I was developing, and the DTraceToolkit.
• Older and out of date Unix or Sun Solaris material is in the Sun Crypt for historic interest.

Videos

Shouting in the Datacenter was a video that Bryan Cantrill and I made on the spur of the moment on New Year's Eve 2008, which went viral (750,000+ views). I've had many emails from companies about it: it seems to be spawning an industry of sound proofing data centers. There is also a making of... video to answer the next question: how did I discover it?

Some of my videos since then, in reverse chronological order:

• At dtrace.conf I gave an unconference-style talk on Visualizations (youtube), summarizing heat maps and Flame Graphs (2012).
• My SCALE10x talk Performance Analysis: new tools and concepts from the cloud is online (2012).
• Short talks on performance tools for Solaris-based operating systems: vmstat, mpstat, and load averages (2011).
• An extended version of my Percona Live New York 2011 talk on Breaking Down MySQL/Percona Query Latency With DTrace was filmed (90 mins).
• A video of my LISA 2010 Visualizations for Performance talk is available on the usenix.org site (80 mins).
• I've given many technical talks of things going right. This talk was about things going wrong: Little Shop of Performance Horrors at FROSUG in Colorado, 2009 (2.5 hours).

Software

These are some small software projects I've developed mostly in my spare time. (I've also written software as a professional kernel engineer, which isn't listed below, including the ZFS L2ARC.) The following are all open source with no warranty - use at your own risk. Some are computer security tools, which may be illegal to own and/or run in your own country if they are misidentified as cracking tools. If you are unsure, please don't download them.

• Dump2PNG visualizes file data as a PNG (uses libpng). An experimental tool intended for core dump analysis. screenshot.
• nicstat prints network interface stats including Kbytes/sec for Solaris (uses Kstat). Example here. There is also a Perl version, and Tim Cook added Linux support.
• Fastburden is a performance testing tool that can generate a flood of client web traffic (multithreaded). Apache or Squid access logs can also be replayed. example.

• FlameGraph: a visualization for sampled stack traces, used for performance analysis (see the posts for Flame Graphs via: DTrace, perf/SystemTap).
• Chaosreader: Trace TCP/UDP sessions and fetch application data from snoop or tcpdump logs. This will fetch telnet sessions, FTP files, HTTP transfers, SMTP emails, ... example output was created by Chaosreader to link to the extracted HTTP sections, telnet sessions, and FTP files found in a snoop log. This can also create telnet replay programs that play back sessions in realtime: example. A tool for forensics or network troubleshooting. download code.
• Perl modules: Net::SnoopLog for snoop packet logs (RFC1761), Net::TcpDumpLog for tcpdump/libpcap logs, Algorithm::Hamming::Perl.
• Distillerror summarizes truss(1) or strace(1) output to highlight errors. See the Solaris example, the Red Hat example, or a larger example.
• FreqCount is a simple frequency counter. Useful for processing logs (most common IP addr, port, etc..). example.
• PortPing is a version of ping that connects using ssh (or other ports), not ICMP. Good for checking firewalls. example.
• MTUfinder tests different sized HTTP requests to a web server, highlighting MTU size problems. example.
• Specials is a collection of "special" programs for system administrators. Mostly Perl.

• DtkshDemos a collection of X11 dtksh scripts. They include xvmstat - a GUI version of vmstat, and xplot - a generic data plotter. Written for any OS with dtksh.
• BBaseline is a small script to create a baseline of the system's performance, by logging the output of several tools. By creating logs during normal and peak activity, this can assist performance tuning. Easy to customize the baseline content, and to grep the baselines. See the example output.
• total is a simple awk script to sum a field (example), and field prints a field (example). These exist for convenience at the shell.

• DTrace Tools such as iosnoop, opensnoop, bitesize.d, execsnoop, shellsnoop, tcpsnoop, ...
• DTraceToolkit a collection of over 200 scripts, with man pages and example files.
• DTraceTazTool a GUI to plot live disk activity.

• K9Toolkit A collection of Perl programs for Solaris that use Sun::Solaris::KStat. This includes tools to print load averages for CPU, memory, disks and network, to aid finding performance bottlenecks.
• FindBill finds backup super blocks on a Solaris UFS for "fsck -o b=..." (if "newfs -N" dosen't help). example.
• listprusage a C program to print process resource usage statistics such as minor faults and syscalls by PID. Solaris (uses procfs). Example here.

• Quick Text Toaster v1.0 An editor I wrote many years ago to grab text from corrupted files. Works with executables, documents, etc.

• QBASIC CRO v1.2 I still find this old program amusing. It is a digital (on/off) CRO that samples the parallel port at 1KHz. screenshot.

• Guessing Game is written in awk C C++ csh Fortran java ksh Pascal Perl QBASIC sh and more as a language comparison. You guess a number between 1 and 100 and your score is saved to high score list. See the example output.
• Sun Crypt has some of my older Solaris and Unix software, including the performance analysis tools Psio for disk I/O by-process, and CacheKit for hardware (I$, D$, E$) and software (DNLC, inode, ...) cache analysis.

Misc

• Recommended Reading: A list of my favourite technology books.
• External Sites: Other interesting places on the web.
• Photos: Some photos I've taken.
• Games: My favourite computer games.