Hard Choices, Tight Timelines: A Closer Look at Skip-level Tradeoff Decisions during Incidents

Unexpected outages in software service delivery - also known as incidents—often require making rapid tradeoff decisions on the road to recovery. Tradeoffs can be relatively minor-—rolling back a recent change or temporarily disabling a certain feature—or they can represent significant threats to reliability or reputation, such as when facing a loss of customer data. While the resolution of incidents is unquestioningly in the hands of engineers, senior management also have an active role in making tradeoff decisions during significant incidents. As researchers interested in software incidents, we recognized a gap in the industry’s understanding of how different levels across the organization work together to resolve challenging incidents.

Our objective in this research is to examine the kinds of tradeoff decisions management faces during incidents, the patterns in how and when they become involved, and the strategies used to coordinate effectively with their incident response teams.

During this talk you’ll get a behind the scenes (and between the ears!) look at management tradeoff decisions and how this knowledge can be used to increase an organization's capacity to handle unexpected events.