Anomaly Detection Based on Job Monitoring Metrics in Distributed System In distributed systems, application delays caused by stragglers become a common problem. And interference by competing the resources can make more stragglers. Previous works mostly focus on straggler detection using statistical analysis methods based on the data extracted from logs. These methods cannot provide finegrained insights to help users optimize their programs. In this paper, we propose an anomaly detection approach using classification method in machine learning based on job monitoring resource metrics. Due to interference, the change of metrics may vary randomly as the job progresses. In order to compare the metrics in different situation, we extract the job, stage and task information from the logs. From the point of system resource utilization, there are three kinds of anomalies we detect, which are the stragglers(tasks), the abnormal jobs and the interfered nodes. We prove that in most situation, more stragglers happen under interference, and the task time for defining stragglers is longer than that in the similar stage time, as well as the node that the abnormal jobs lived is the interfered nodes.We use the task time in the same stage to label the data for training the adaptive boosting classifier model solely with the resource features. In this way, the model can detect straggles, abnormal jobs and interfered nodes in realtime. Additionally, Experiments show that the accuracy of anomaly detection reaches 92%. Case studies show that our framework is effective in detecting abnormal jobs and interfered nodes.

ALSR: An Adaptive Label Screening and Relearning Approach for Anomaly Detection Anomaly detection using KPIs (Key Performance Indicators) is key to AIOps (Artificial Intelligence for IT Operations). Recent anomaly detection approaches have adopted machine learning to detect anomalies on the perspective of individual time points more than events. These approaches do not make effective use of the labels of continuous anomaly intervals, nor do they pay attention to the differences among anomaly points. The detection performance is therefore not high enough. In this paper, we propose an anomaly detection approach named ALSR, which uses a label screening model and a relearning model to analyze and utilize the continuous anomaly intervals of KPIs in finer granularity. The label screening algorithm takes advantage of the continuity of anomaly intervals to remove unnecessary data from the training set, so as to better suit to intervaloriented anomaly detection. The relearning algorithm reclassifies the truefalse positive points within range of detected anomalies, thus effectively reduces the number of false positive points. ALSR uses statistical characteristics and time series models for feature extraction, and the feature set is proved to better describe the characteristics of KPIs. We conduct comprehensive experiments on 25 KPIs, and the total Fscore of ALSR is 0.965, which outperforms stateoftheart anomaly detection approaches.