SREcon21 Conference Program

How do you reconcile the ideal of blamelessness with the demand for blame? When is it constructive to hold someone accountable, and how? To change a blameful culture, we must empathize with those that point the finger and see how their goals align with our own. We'll show you how to communicate that their goals can be achieved blamelessly. Lastly, we'll share how to hold true accountability well.

This talk is based on a real journey on establishing SLOs for an infrastructure SRE team whose availability target is higher than 99.999%. First, we reveal our process on defining SLOs and demonstrate the gaps between expectations and reality on using SLOs with dev teams. Secondly, we present a uniformed SLO framework (SLX) design to facilitate SREs to manage hundreds of SLOs. For example, other than using SLO data for basic alerting and weekly reporting, we combine the SLO framework with statistical anomaly detection algorithms to locate the pitfalls automatically. To achieve that, we introduce several new concepts like Service-Level-Factor (SLF) and Service-Level-Dependency (SLD) and use them to build SLO knowledge graphs across multiple infrastructure systems. Finally, we present our intent-driven SLX implementation inspired by the Kubernetes design and the Gitops paradigm.

We ingest over a trillion events per day into our cloud platform and it is very important that this platform is available, operational, reliable, and maintainable.

In creating a comprehensive monitoring strategy for our data processing platform, we found it strategic to model our platform's efficiency and resilience along two axes—complexity of implementation and engineer experience—from which we can define four quadrants—observability, operability, availability, and quality.

In this talk, we present how we've employed this four-quadrant model to establish key indicators and enforceable quality SLAs in order to improve the resilience of our cloud platform while reducing operational complexity.

Whenever a company scales quickly, they invariably take on technical debt. It's unavoidable. However, the existence of tech debt isn't the problem, it's how you handle it when you can't put it off any longer.

In this talk, I will be telling the story of one company's largest technical re-architecture to date. It was a company-wide effort that extended over multiple years and taught us many lessons. And it all revolved around a single, overloaded database.

This talk is geared towards mid-level engineers and up who have a solid understanding of tech debt and distributed systems. I discuss some advanced topics and show various architectural diagrams. Though beginners will likely it interesting as well.

March 2020 was a strange month for everyone—our work and employee interactions changed fundamentally, and perhaps permanently, as the entire office-bound workforce shifted to working from home. Here in Aspiegel, it wasn't the only challenge that came our way. We combined an increased role in Huawei service management, with retiring our managed services SRE team. This meant that over the year we would need to hire aggressively to replace the team, and also to support our new growth. Working through this onboarding over the course of the year would cause some hiccups along the way, but ultimately it would force us to change into a leaner, and more professional SRE Department. Join us, as we talk about what we did, what we learned, and how we can help others get there too!

Many teams will have practiced and refined their Incident Management skills and practices over time, but no one had a playbook ready to go to manage the dramatic Coronavirus-driven usage growth of Google Meet without a user-facing incident. The response resembled more a temporary reorganization of more than 100 people than it did your typical page—the fact that there was no user-facing outage (yet), notwithstanding.

This talk will cover what this incident response structure looked like, and what made it successful.

Many SREs treat the idea of AIOps as a joke, and the community has good reasons for this. But what if it were actually true? What if our jobs were in danger? What if AI—which can play chess, Go, and Breakout more fluidly than any human being—was poised to conquer the world of production engineering as well? What could we, or should we, do about it?

Join me in this talk as we examine the current state of affairs, and the future, in the light of the promises made by AIOps companies. Or, in short, to ask the question, what if AIOps were true?

In many ways, incident management is the "emergency room" for technical systems. As technology has evolved, it has progressed from auxiliary systems, to essential business systems of record, to critical systems of engagement across multiple industries. As these systems become increasingly critical, SRE's role in incident management and resolution has become vital for any essential technical system. This talk focuses on how various strategies used in the medical field can be applied to incident response. From looking at algorithm guided decisions (and learning a bit about what "code blue" really means) to discussing approaches to triage and stabilization based on the ATLS protocol, to considering the role of response standardization such as surgical checklists in reducing cognitive overhead (especially when PagerDuty goes off at 2 a.m.!), this talk aims to take key learnings from the medical field and apply it in practical ways to incident management and response. This talk is largely conceptual in nature, with takeaways for attendees from a wide variety of backgrounds and technical experience levels.

Would it surprise you to know that time-series data dates back hundreds of years to the early days of science, statistics, and data collection? It shouldn't when you think about it and there's plenty of good lessons learned we can re-apply to our dashboards and other engineering presentations today.

"Improving Observability In Your Observability" will present practical takeaway lessons that engineers can immediately utilize across various practices and at all levels to improve the visual understanding and credibility of their informational presentations of metrics and observability data.

As SREs interested in better presenting our telemetry we'll explore what practical lessons we can learn from the works of Galileo, Charles Joesph Minard, and Edward Tufte as well as simple pitfalls to avoid and what we can do to improve the transfer and content density of our dashboards and other engineering graphics.