The Secret Lives of SREs - Controlling the Costs of Coordination across Remote Teams

If you ask a group of engineers how they resolved a particularly difficult outage they typically talk about the dashboards that got pulled up, the logs they looked at, the node someone restarted, or the jobs they killed on their way to restoring the service. But that doesn't do much to tell us how, given conditions of uncertainty and time pressure, practitioners flexibly apply their knowledge to novel problems.

In other words, how did an engineer know what was the 'right' thing to do in spite of ambiguous data or who was the 'right' person to help diagnose a particularly out-of-control problem?

Over the last 3 years, as part of her dissertation work, Dr. Maguire studied both established and ad hoc teams of engineers responding in real-time to service outages ranging from minor disruptions to potentially organizational-viability-crushing events. In her research, she examined 62 cases of incident response across 4 organizations of varying scale and complexity to understand how engineering teams manage these costs of coordination under differing circumstances.

In this talk, Dr. Maguire will highlight some surprising (and provocative!) findings such as:

• Incident management works very differently than existing domain models (like GoogleSRE) suggest and incident command can actually be counterproductive to fast resolution;
• The choreography of the cognitive work in this joint activity is shown much more subtle and highly integrated into the technical efforts of dynamic fault management than previously understood;
• Tooling designed to aid coordination can incur additional cognitive costs for practitioners;
• Strategies of 'adaptive choreography' enable practitioners to cope with dynamic events and dynamic coordination demands;
• How tooling and intra-organizational dependencies can shift costs of coordination across time and organizational boundaries, increasing complexity for SREs.