Cross-System Interaction Failures: Don't Fail through the Cracks

Modern cloud systems are orchestrated by many independent and interacting subsystems, each specializing in important services such as scheduling, data processing and storage, resource management, etc. Hence, overall cloud system reliability is determined not only by the reliability of each individual subsystem, but also by the interactions between them. With recent practices of microservice and serverless architectures, each individual subsystem becomes simpler and smaller, while their interactions grow in complexity and diversity. We observe that many recent production incidents of large-scale cloud systems are manifested through failures of the interactions across system boundaries, which we term "cross-system interaction failures", or "CSI failures". However, understanding and addressing such failures requires new techniques and practices that are often unavailable or under-developed.

In this talk, we will present our recent work on understanding and preventing cross-system interaction failures. Specifically, we will characterize the many faces of cross-system interactions and their diverse failure modes in modern cloud-native system stacks. We will then discuss the gaps in the current practices and existing techniques in the context of software testing and verification. Lastly, we will present some of the new techniques we developed at the University of Illinois at Urbana-Champaign to address cross-system interaction failures.