Complex distributed Internet services form the basis not only of e-commerce but increasingly of mission-critical networkbased applications. What is new is that the workload and internal architecture of three-tier enterprise applications presents the opportunity for a new approach to keeping them running in the face of many common recoverable failures. The core of the approach is anomaly detection and localization based on statistical machine learning techniques. Unlike previous approaches, we propose anomaly detection and pattern mining not only for operational statistics such as mean response time, but also for structural behaviors of the system-what parts of the system, in what combinations, are being exercised in response to different kinds of external stimuli. In addition, rather than building baseline models a priori, we extract them by observing the behavior of the system over a short period of time during normal operation. We explain the necessary underlying assumptions and why they can be realized by systems research, report on some early successes using the approach, describe benefits of the approach that make it competitive as a path toward self-managing systems, and outline some research challenges. Our hope is that this approach will enable "new science" in the design of self-managing systems by allowing the rapid and widespread application of statistical learning theory techniques (SLT) to problems of system dependability. A "five nines" availability service (99.999% uptime) can be down only five minutes a year. Putting a human in the critical path to recovery would expend that entire budget on a single incident, hence the increasing interest in selfmanaging or so-called "autonomic" systems. Although there is extensive literature on statistics-based change point detection [2], some kinds of partial failures, or "brown-outs" in which only part of a service malfunctions, cannot be easily detected by such techniques. For example, one of the auPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. thors experienced a bug such that after clicking to purchase a flight for April, a later visit to the "flight details" page showed the wrong flight date (in October) and no flight itinerary details at all. If the operational statistics such as response time for delivering this page are within normal thresholds, performance monitoring would not find this problem. Indeed, such "glitches" accounted for up to 64% of user-visible failures in the 40 top-performing Web sites according to an industry group survey [4] , and our own anecdotal experience with one large service provider reveals that 75% of their total recovery time for application-level failures is spent detecting them, and another 18% diagnosing them [9] . We believe a promising direction is to start thinking not in terms of normal operation vs. recovery, but in terms of constant and rapid adaptation to external conditions, including sudden workload changes, inevitable hardware and software failures, and human operator and programmer errors. In particular, we propose the broad application of techniques from statistical learning theory (SLT)-automatic classification, novelty/anomaly detection, data clustering, etc.-to observe and track structural behaviors of the system, and to detect potential problems such as the example above. We assume typical request-reply based Internet services, with separate session state [18] used to synthesize more complex interactions from a sequence of otherwise stateless request-reply pairs. Past approaches to statistical monitoring of such services have primarily relied on a priori construction of a system model for fault detection and analysis; this construction is tedious and error-prone, and will likely remain so as our services continue to evolve in the direction of heterogeneous systems of black boxes, with subsystems such as Web servers, application logic servers, and databases being supplied by different vendors and evolving independently. We propose instead to build and periodically update the baseline model by observing the system's own "normal" behavior. The approach can be summarized as follows: 1. Ensure the system is in a state in which it is mostly doing the right thing most of the time, according to simple and well-understood external indicators. 2. Collect observations about the system's behavior during this time to build one or more baseline models 49
Fox et al. proposed an anomaly detection approach that considers both simple operational statistics and structural change in a complex distributed system REF .
Combining statistical monitoring and predictable recovery for self-management
WOSS '04
Computer Science