Taming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb

Logs, much like currency, are subject to decreasing value over time. Observability teams face the challenge of ensuring high availability of recent logs, especially during incidents or deployments. Traditional log search systems struggle to auto scale cost-effectively and respond fast enough during an incident. In this session, we will discuss how Slack tackles spiky log volumes in ElasticSearch, from detecting log spikes at various stack layers to handling them using rate limiting, quotas, sampling, and back fills.

While these techniques help, they may result in data loss. Therefore, we will delve into the automation of handling log spikes using Kaldb, an open-source log search engine. We will explore trade-offs to minimize data loss, such as prioritizing the ingestion of fresh data over older data while auto-scaling. Powered by Lucene and OpenSearch, Kaldb allows Slack to prioritize fresh log data and rapidly scale capacity within a Kubernetes-based architecture.