From Chaos to Clarity: Deciphering Cache Inconsistencies in a Distributed Environment

In this session, we'll discuss a distributed caching system used at Netflix for streaming, live, games, Ads etc in multiple regions on a public cloud. There are various components that make the system highly resilient - control plane, replication engine, proxy, cache warmer etc. In this session, we will touch up on replication engine which processes around 30 million requests per second and manages to keep the response times for 95% of these requests under 2 seconds across the regions. We present and deep dive into a problem that almost put us at the risk of delaying the global launch of a business critical initiative. We will walk you through the entire process from the start to finish, the debugging journey, our takeaways and how these debugging techniques are generally applicable to any organization. The talk will focus on how and when problems get introduced, how can a simple assumption break the entire stack and how can these be caught sooner.