Kube, Where’s My Metrics? The Challenges of Scaling Multi-Cluster Prometheus

Service and systems monitoring is crucial for healthy, happy applications. For small to medium-sized Kubernetes environments, a single Prometheus HA pair is often sufficient. But when you scale your environments, new challenges arise. As Prometheus ingests and stores significantly more time-series metrics, it can struggle to keep up. And if cluster numbers increase, it becomes difficult to locate the metrics you need. These are known problems with a few common solutions.

In this talk, Niko and Iain will discuss the challenges of scaling Prometheus by walking you through the history of monitoring Grafana Labs' internal services. As they scaled to 40+ clusters over five cloud providers, the Grafana Labs teams frequently iterated on their internal monitoring architecture. They'll cover the basics of Prometheus for monitoring and alerting before moving into remote write and monitoring of multiple clusters.