Brendan Gregg explains how to use short checklists and quick procedures to analyze and solve complex systems performance issues under pressure.
Brendan Gregg, Netflix
There's limited time for performance analysis in the emergency room. When there is a performance-related site outage, the SRE team must analyze and solve complex performance issues as quickly as possible, and under pressure. Many performance tools and techniques are designed for a different environment: an engineer analyzing their system over the course of hours or days, and given time to try dozens of tools: profilers, tracers, monitoring tools, benchmarks, as well as different tunings and configurations. But when Netflix is down, minutes matter, and there's little time for such traditional systems analysis. As with aviation emergencies, short checklists and quick procedures can be applied by the on-call SRE staff to help solve performance issues as quickly as possible.
Why it's here
2 people recommend this
What people say. Recommended as a guide for creating incident response runbooks and performance checklists.
p
u/procesdin r/sre • Recommends
On “Incident response katas”
Documentation comes with a initial and a maintenance cost so not all will get a runbook but for special circustances
* severe incidents that can arrive to the on-call person (at 3:00AM , from an env where they may have never worked before)
* very complex known issues where the investigation was waaay to long to repeat
we tried to define runbooks/checklists similar to the ones described here Brendan Gregg - Performance Checklists for SREs
