SREcon17 Asia/Australia Conference Program

TechLadies is a community-led initiative for women in Asia to connect, learn, and advance as programmers in the tech industry. Since our launch in 2016, we’ve taught 177 ladies in Singapore and Malaysia programming skills and saw 7 ladies getting technical internships or hired as junior software engineers. In this talk, I’ll share about the inspiration behind setting up TechLadies, the what and how of the programs we do, and the lessons learned on how to introduce more women to the tech industry. Hopefully it will be useful for you in increasing gender diversity at your workplace!

Anomaly detection based on time series analysis approaches has been a focused theme in the field of monitoring, especially for business indicators monitoring. In Alibaba, hundreds of major KPIs need to be monitored in real time to detect the abnormal events and raise alarms. The effectiveness of the alarms is strictly evaluated by human operators. Therefore, we proposed an intelligent anomaly detection method to make the business monitoring system more scalable and easier to maintain.

There are two major problems in traditional anomaly detection approaches:

• How to get accurate predictions based on seasonal time series data with complex noises and interferences.
• How to determine the segmental thresholds dynamically, and learn from human feedbacks, e.g. the manual labelling data, to improve the accuracy of anomaly detection, and tolerate the error, event contradictions within the labelling data. 

For the first problem, as a tradeoff among computation performance, accuracy and robustness, we introduced a specific method to pre-process the data, and chose the STL method to do predictions on our business data.

For the second problem, we proposed a closed loop feedback method to determine the initial segment thresholds, and utilized the human labelling data to update the thresholds continuously.

There are already many outstanding open-source monitoring systems. However, with rapid growing businesses and specific requirements of internet companies, the existing open-source monitoring systems are not better enough on performance and scalability. 

Under the situation, Open-Falcon’s main distinguishing features compared to other monitoring system are: 

• Scalability: Scalable monitoring system is necessary to support rapid business growth. Each module of Open-Falcon is super easy to scale horizontally. 
• Performance: With RRA(Round Robin Archive) mechanism, the one-year history data of 100+ metrics could be returned in just one second. 
• High Availability: No criticle single point of failure, easy to operate and deploy. 
• Flexibility: Falcon-agent has already 400+ built-in server metrics. Users can collect their customized metrics by writing plugins or just simply run a script/program to relay metrics to falcon-agent. 
• Efficiency: For easier management of alerting rules, Open-Falcon supports strategy templating, inheritance, and multiple alerting method, and callback for recovery. 
• User-Friendly Dashboard: Open-Falcon could present multidimensional graph, including user-defined dashboard/screen.

Monitoring system is an important tool for SREs to guarantee service stability and availability. Baidu’s monitoring system, Argus, keeps tracking hundreds of services of distribution system. With the increasing complexity of the services, the monitoring items and the corresponding anomaly detectors grows to a magnitude that the generated alerts floods from time to time. On an average day, Argus detects millions of warning events and sends out thousands of SMS alerts to on-call engineers. This results in a per person amount of more than 100 alerts during the day time and 30 during the night time. When a severe failure occurs, the alerts flood in a massive surge, and become little helpful for the engineers to fix the problem. Therefore, it is imperative to improve the detection accuracy of abnormal events, reduce the amount of alerts, and organize them into a meaningful way.

In this talk, we will introduce our practice that leverage machine learning methods to detect anomalies and group alerts, in order to solve the above issues. We will also share some successful experiences, such as alert based datacenter-level failure detection, and alert-triggering automatic recovery techniques.

When an incident happens it's the worst possible time to be bogged down with confusing systems and processes. A well defined Incident Management Process that's light-weight and supported by good automation offers a way to get fast, easy and predictable results during an incident, but if you don't implement the right things in the right way you risk bad results, such as high time-to-recovery, at critical times.

Find out how Atlassian drives value out of the Incident Management process and what metrics we use to track it. We'll also cover how we created automation to remove the overhead in managing incidents and deep dive into a case study to explain how it all ties together.

The target audience for this conceptual session is people who are involved in the management of incidents, such as SREs and delivery team members.

LinkedIn has hundreds of microservices spanning across different data centres and dependent on each other. Even though microservice architecture has its own advantages, often times, identifying the problematic service during an incident/outage could significantly contribute to a high MTTR and unnecessary escalations. Event Correlation Engine is an attempt to algorithmically identify the responsible service quickly and escalate to the right team.

Event Correlation Engine examines the entire service stack by looking at critical downstream latencies, error metrics and other monitoring events to recommend a responsible service. It understands callee caller communication for linkedIn’s complex microservices architecture.The engine uses dynamic thresholds, which it learns by processing last 30 days data, to provide an effective recommendation. We’ll discuss the approach we used in building it and how it is being used at LinkedIn to reduce MTTR and on call escalations.

Large Internet products are too complex to completely recover from a failure rapidly, as root cause localization and large scale operation are very time-consuming. If the failures can be isolated to a small part of system, we can transfer user query to the other parts still work, or cut off the failed minor subsystem, which is much more rapid than completely system recovery. This talk will presents some practical experiences for failure isolation in Baidu.

First, we should have at least N+1 data center redundancy, and eliminate unnecessary global “single-point” module. All automated operations should be limited to execute in one data center first. When the system in one data center is damaged by network or software failure, we can transfer user requests to other data centers rapidly, even automatically.

Second, we make the non-essential components of the system detachable. When one of them fails, it can be detached immediately to keep the major function still work for user.