Jia Guo, LinkedIn
Apache Pinot, a real-time, distributed OLAP database, encountered resiliency challenges in its large production deployment due to server failures, slowness, and unpredictable query patterns. To ensure adherence to strict SLAs, enhancements were made to its Query Processing framework through two key features: Intelligent Query Routing and Runtime Query Killing.
Intelligent Query Routing replaces the traditional round-robin server query distribution with a dynamic approach based on continuous server performance metrics. This adaptive method proactively routes queries to more efficient servers, reducing the need for manual intervention and preserving availability during transient server issues.
Runtime Query Killing addresses the issue of complex queries causing Out of Memory errors and availability dips. By monitoring resource consumption in real-time and employing a lockless sampling algorithm for minimal overhead, this framework can terminate resource-intensive queries preemptively, safeguarding system availability and maintaining high SLAs.
This technical talk will delve into the design, deployment practices, and observed benefits of these features in a large-scale Pinot deployment, offering insights for developers and administrators of similar large-scale distributed data management systems.
Jia Guo, LinkedIn
Jia Guo is a System and Infrastructure Engineer at LinkedIn's Online Analytics team, focusing on Pinot query execution, performance, and reliability. Prior to LinkedIn, Jia specialized in advancing computing systems through his academic research, and received his PhD in Distributed and Parallel Computing from The Ohio State University.
author = {Jia Guo},
title = {Strengthening Apache Pinot{\textquoteright}s Query Processing Engine with Adaptive Server Selection and Runtime Query Killing},
year = {2024},
address = {San Francisco, CA},
publisher = {USENIX Association},
month = mar
}