Reader Aids - Abstract -Most error log analysis studies perform a statistical fit to the data assuming a single underlying error process. This paper presents the results of an analysis that demonstrates the log is composed of at least two error processes: transient and intermittent. The mixing of data from multiple processes requires many more events to verify a hypothesis using traditional statistical analysis. Based on the shape of the interarrival time function of the intermittent errors observed from actual error logs, a failure prediction heuristic, the Dispersion Frame Technique (DFT), is developed. The DFT was implemented in a distributed on-line monitoring and predictive diagnostic system for the campus-wide Andrew file system at Carnegie Mellon University. Data collected from 13 file servers over a 22 month period were analyzed using both the DFT and conventional statistical methods. It is shown that the DFT can extract intermittent errors from the error log and uses only one f i h of the error log entry points required by statistical methods for failure prediction. The DFT achieved a 93.7% success rate in failure prediction of both electromechanical and electronic devices. Trend analysis utilizes the general system behavior as captured in the system error files for fault diagnosis and failure prediction. The basic hypothesis is that there exists a period of instability prior to a hard failure. Work at CMU [8] has demonstrated the feasibility of this hypothesis from observing clusters of disk errors on DEC computers, where the interarrival times of error events were shown to decrease prior to initiation of the repair action. Tsao's research focused on the development of the tuple concept as an information organizing and data reduction technique, since multiple reporting of errors was often the first obstacle in data analysis. Another approach to organizing information was developed at the University of Illinois to characterize the relationship among errors recorded in a system error log [2] . A probabilistic model was used to automatically detect symptoms of frequently occurring persistent errors in two large Control Data Cyber systems. It was shown that 85 % of the identified error symptoms, typically composed of over 30 error events ', corresponded to permanent system faults. The concept of observing system trends €or failure prediction was further investigated at Stanford [6] . A methodology that involved three types of analysis was outlined: an average error distribution for each error type, an error distribution for all error types, and a failure/CPU utilization relationship. Both individual and total error distributions demonstrated increasing error generation rates prior to a system crash. A failure prediction algorithm based on the detection of large error clusters, that is, a threshold number of errors, was proposed. Preliminary results using an average of 113 errors for each prediction over six months of data were analyzed showing a 60% chance of success. Although this method has neither been thoroughly tested nor implemented, at least the results clearly indicated that failure prediction based on an increase in error rate, a threshold error number, a CPU utilization threshold, or a combination of these factors may be feasible. Most prior work has modeled error logs as if they were produced by a single error source. Statistical methods are employed to estimate parameters and identify the error source. Experience with error logs suggest that they intermix entries from a number of error sources. If the error log is analyzed for events such as system crashes (which have a multitude of causes), more data is required to reach a statistically meaningful conclusion than if the events were sorted into contributory causes and analyzed separately. Furthermore, conclusions drawn from analysis of multiple error sources that have been commingled are more difficult to generalize and to apply to new systems. The paper is divided into five sections. Section 2 describes the experimental data collection system. The data was used in two studies: to determine the statistical characteristics of intermittent and transient errors in section 3 and to validate a predictive trend analysis heuristics for intermittent errors in section 4. Section 3 develops a methodology for separating an error log into independent sources. The separation methodology is based upon the fact that intermittent errors reoccur, often at an increasing rate. In section 3.2, traditional statistical analysis methods are applied to each intermittent error source showing that at least 25 errors spanning up to 18 months are required to identify the trend. In section 3.3, errors not associated with an intermittent error source are assigned to a single transient error source to which traditional statistical analysis are also applied, yielding Weibull functions whose shape parameters are ' In one example shown in the paper, a symptom is extracted from three events with a total of 36 error records. less than 1 .O. Next section 3.4 applies traditional statistical analysis to the total error log (ie, all intermittent and transient error sources) and compares the parameters extracted to those derived from analyzing an artificial error log formed from combining two pure error sources. Similarities between the actual and artificial error logs supports the concept that actual error logs are composed of entries from multiple error sources. Thus variations between error logs and within an error log over time are a function of the relative contributions from each pure error source. A good working hypothesis is that a single transient and a single intermittent error source exist in the error log at any given time. Based on the experience gained in factoring error logs into individual error sources as well as from interviews with maintenance personal, a new heuristic, the Dispersion Frame Technique (DFT) is introduced in section 4. The DFT extracts error log entries caused by individual intermittent faults, and then applies one of its five failure prediction rules according to the interarrival patterns of the errors. The five rules are shown to capture behavior corresponding to that detected by traditional statistical analysis techniques. Moreover, the DFT used only one fifth of the error log entries required by statistical methods, and achieved a 93.7 % success rate in failure prediction for both electromechanical and electronic devices when applied to 22 months of error log data from the Carnegie Mellon University (CMU) campus-wide Andrew file system. Finally, section 5 concludes the paper. In particular, it summarizes previous work on trend analysis including both statistical methods using the Weibull distribution and the DFT as shown in table 1. The table lists the approach, the average number of events required to identify a trend, and the percentage of success in applying the approach. Although the data sets adopted from individual sources are different, it nevertheless shows that the DFT uses the least number of data points given similar error event recording techniques.  sists of personal workstations with raster graphics, high bandwidth networks, and a time-sharing file system called VICE. Currently there are 13 file servers in the VICE file system. The file server hardware is composed of a SUN 2/170 (or SUN 3/280) workstation with a Motorola 68010 (or 68020) microprocessor, a Xylogics 450 disk controller and Fujitsu Eagle disk drives each accommodating up to 800 Megabytes. A file server has at most two disk controllers and each controller supports up to two disk drives. According to published statistics, the Mean Time Between Failure (MTBF) of the disk drive is 20 OOO hours. Therefore when the file system is run at full capacity with 52 disk drives spread over 13 file servers, one could expect a hard disk failure in the file system every 400 hours2. The load on the file servers was empirically determined to be constant with very little variation over time. Thus one of the variables (workload) previously found to affect error rate, could be removed as a factor resulting in a simplified analysis. In order to support error collection and analysis, the device driver and the error handling software in the SUN UNIX kernel was modified. This software instrumentation enables logging of error messages prior to system crash and collecting the relevant information in each kernel error report in a uniform format for analysis. Moreover, an on-line monitoring and predictive diagnostic system composed of two major pieces of software, the Agent and the Diagserver, was developed for the VICE File System. The Agent, residing in the node under observation, is in charge of kernel message acquisition. The Diugserver, residing in a centralized site, performs data analysis for failure prediction and diagnosis. This paper concentrates on the issues and techniques in failure data analysis and prediction. A detailed list of the kernel changes, and the architecture of the on-line monitoring and predictive diagnostic system can be found in [4] . Sources of information for data analysis include the automatic error log of 13 VICE file servers, collected by the on-line predictive diagnostic system, and an operator's log. Data collected from February 1986, the first date of file server operation, until January 1988, was used for studying the characteristics of permanent, transient, and intermittent faults. The increasing number of file servers placed in service over the twenty-two month period resulted in a total of 20 workstation-years of data. Errors are considered to be manifestations of faults. Thus the error log contains errors whose causes are faults. It is shown that the typical error log contains events that are caused by a mixture of transient and intermittent faults. The operator's log contains permanent failure information as well as repair actions attempted to remedy the problem. The terms intermittent and transient have been used interchangeably Andrew is the software system consisting of four major components that supports the Carnegie Mellon University largescale distributed computing environment. The environment con2However the observed file system disk mean time between failure is 1671 hours and the system mean time to crash due to all hard failures is 504 hours [Lin 881.  LIN/SIEWIOREK: ERROR LOG ANALYSIS: STATISTICAL MODELING AND HEURISTIC TREND ANALYSIS The failure distribution mathematically characterizes the probability of system failures as a function of time. The exponential, gamma, Weibull and lognormal are all well known distributions in failure analysis [5] . The Weibull function is used in this research and is defined as: where a is the shape parameter, and X is the scale parameter. Note that when a equals 1, the distribution function reduces to the exponential: The hazard function is the time-varying failure rate. The Weibull hazard function is defined as: The shape parameter a directly influences the hazard function as follows: If CY c 1 , the hazard function is decreasing with time; If a = 1, the hazard function is constant with time, ie, the If a > 1, the hazard function is increasing with time. It has been suggested that transient faults are characterized by a < 1, permanent faults by a = 1, and intermittent faults by a > 1 [7] . In our study of permanent faults (ie, those obtained from the operator's log that involve actual repairs), the statistics showed an estimate of a to be 0.92 and the data followed an exponential distribution within 0.05 level of significance [4] . In this paper, we will show that while a < 1 is appropriate for transients, that a > 1 is an oversimplification for intermittent faults. However, we must first develop a methodology for separating the error log into its constituent error sources. exponential distribution; Since intermittent faults are repairable by replacement, they are associated with a physical subsystem, sometimes referred to as a field replaceable unit (FRU). Error log entries are identified by the hardware or software error detection mechanism which reported the error. For each FRU a timeline was constructed, composed of only those error log entries identifying the FRU. Since a FRU may be repaired several times during the course of the study, a heuristic was required to differentiate between two independent intermittent faults. The time based clustering heuristic previously used to collapsing multiple error log events into a single logical event [8] was extended to collapse multiple intermittent errors into a single physical failure. The system operator's log was used to identify the repair activities which culminated in insertion of a new FRU. There were 29 repairs of hardware failures in the 22 months of observation including 7 disks, 7 CPUs, 7 memory boards, and 8 disk controllers. If the interarrival time between errors on a FRU's timeline were more than a week apart (168 hours), the errors were considered to be unrelated and the growth of the timeline was terminated. Error events left on a timeline were called intermittent while all the rest were called transient. The threshold of 168 hours was chosen based on two observations. First, in the study of transient errors, discussed in section 3.3, the average interarrival time for transients regardless of FRU was found to be 354 hours over all systems, and the smallest mean observed in an individual system was 154 hours. The transient data was analyzed for several thresholds between 154 to 354, but all resulted in the same Weibull parameters. Second, in order to determine a single threshold, 168 hours was chosen to avoid potential cyclic patterns due to daily dependencies on workload. The timelines of 16 of the 29 repair actions and their corresponding intermittent errors are shown in [4]. The remaining 13 had fewer than five error entries, too few for analysis. Four of these timelines are reproduced in figure 1 with ' + ' representing device errors and 'A' indicating device repair. Periods of increasing error rate, which appear as either clusters of errors or decreasing interarrival times between errors (suggesting a Weibull failure distribution with a > l), are observed. Although the clustering patterns shown in figure 1 spanned about 200 hours, the majority of the 16 failures which recorded error log activity were preceded by error log entries over lo00 hours prior to repair.  Modeling of intermittent faults begins with the analysis of the interarrival times of their manifestations (eg, errors) [7] . This is done by calculating the difference between the time stamp information of each intermittent error and by formulating the hazard function to identify the associated distribution function. A linear regression analysis was performed on the interarrival time data to estimate the maximum likelihood estimates for the Weibull parameters. The technique is based upon the transformation of the Weibull cumulative distribution function into a linear function of In (t) , where the initial elements (Y and X are obtained from the slope and the Y-intercept of the straight line. The linear estimates can be used as initial values for an iterative Newton-Raphson solution method to obtain the maximum likelihood estimates of (YML and XML. Subsequently, a chisquare goodness of fit is performed to evaluate the fit of the Weibull to the observed data. A significance level of 0.05 was selected, which means the probability that a chi-square random variable with m degrees of freedom, where m is calculated as the number of categories3 minus the number of parameters to be estimated minus 1, will exceed x,' is c (ie, 0.05). Categories are chosen such that the expected observations in each category is no less than 4. Note 1. Repair actions 7&8 occurred within a short period of time. The disk controller was replaced due to a string of disk errors. However, xy2 disk was lost at power up. Therefore data points were accounted for both repairs. repair. Second, among the 17 Weibull shape parameters (aML) estimates made, six are greater than 1 (Vice2.xyO-2, Vicd.cpu, Vice7.cpu, Vice9.mem-2, Vice12.xy2-1, and Vicel4.cpu), three are close to 1 (Vice2.men, Vice3.con, and Vice8.cpu), and eight are less than 1 [4]. Thus simply looking for a greater than 1 is insufficient to identify the trend of an intermittent fault. Third, the statistical analysis shows that most of the repair action were performed before system statistical trends developed, indicating that users do not tolerate the large number of errors required for a statistical method to predict a failure trend. Therefore a new method as described in section 4 was sought for trend analysis and failure prediction. Transient errors were extracted from the system event log by subtracting known hard-failure-induced intermittent errors as described in the previous section. Similar to the intermittent errors, the transient errors were a combination from several pure error sources. The data exhibited three types of the most commonly seen transient errors: system software errors (event type SOFT), parity errors (event type MEM), and unscheduled system reboots (eg, watchdog resets). Each type of transient error could have been composed from multiple pure error sources but there was insufficient information associated with the error types to further separate them into unique error sources. Hence only the total population of transient errors was modeled. Modeling of transient errors begins with the analysis of their interarrival time [7] . The interarrival times are calculated using the time stamp information from the system event log. The hazard function is formulated and analyzed to identify its associated reliability function. In total, 446 transient errors4 are plotted in figure 2 . The x-axis divides the interarrival times into 20-hour bins, while the y-axis shows the number of occurrences in each bin. The obvious skew toward the low end for all the data indicates that the Weibull distribution is a likely candidate for the reliability function. Table 3 shows the shape and scale parameters of the transient errors for the thirteen file servers. Its format is identical to that of table 2 except that the FRU column is replaced by the "total" time in hours each file server was under observation. In order to perform the chi-square test, at least four categories are required for a Weibull and three for an exponential fit. Rows marked with an asterisk represent those file servers whose number of categories were insufficient to perform the chi-square goodness-of-fit tests, implying their estimates of the parameters are inconclusive. Even so, only one out of the four asterisk-marked rows has an a greater than one. The (YML values for all the other servers are less than one. The underlined chi-square values failed the chi-square test. In fitting the data to the Weibull function, VICE2 is the only server that failed 4The total number of system crashes due to transient was 269, accounting for 90% of the total number of system crashes. Table 3 indicates that a minimum of 25 error points spanning up to 18 months are required to gain a useful estimate of the parameters of the function. In addition, the assumption of near constant load on the file servers was tested by sampling the load every 53 hours (ie, the system mean time to crash). The system usage was found to be uniform, thus verifying the assumption that system load could be factored out as a variable in this study. As we have seen, the error log is a mixture of both transient and intermittent errors. The arrival process of transient errors was shown to have a decreasing failure rate while intermittent errors tend to have a constant or increasing failure rate. A similar observation was found in a study by [McConnel 791 where the a of the Weibull distribution for parity errors (ie, transients) was 0.5 while the a for all crashes (mixture of transient and intermittent errors) was closer to 1 .O with a value of 0.8. In order to study the impact of mixing two error sources in more detail, a simulation study was conducted and compared with the experimental data. The same statistical modeling process was performed on the entire error log data with intermittents and transients intermixed. Table 4 shows the parameters of the entire error log for each file server assuming a Weibull distribution with a, and A,. The row marked with an asterisk indicates that the data from VICE 13 is insufficient to perform the chi-square goodness-offit test, implying its estimates of the parameters are inconclusive. The underlined chi-square values denote those that failed the chisquare goodness-of-fit test assuming a 0.05 level of significance. VICE6 passed the chi-square test with a 0.25 level of confidence, VICE9 with a 0.1, and VICE12 with 0.95, which is an unrealistic significance level. It can be seen that (;YML values for all the servers are less than one. The effect of intermittent faults on the entire error log can be observed in the parameter changes between tables 3 and 4, and is summarized in table 5. In table 5, it can be seen that given a, < 1 for transient faults, ai of intermittent faults directly influences a, of the entire error log. The underlined values denote inconclusive results. The amount of the shift Aa (ie, I a,-ar I) is a function of the ratio of the shape parameters a,/ai and the relative number of occurrences of errors N,INi, where N denotes the number of error occurrences. Therefore, if transient errors cannot be isolated from the system error log, the resulting statistical analysis of the entire error log to identify the presence of an intermittent fault will require more data points than for intermittent data alone. Furthermore the analysis is less likely to be conclusive. The effect of ai on a, is summarized in table 6 . In general, if ai is less than 1, a, < a,; if ai is close to 1 (+O.l), a, > a,, and if ai is greater than 1, a, > a,. However these observations do not hold in all cases (eg, VICE7 and 8 in table 5). Since the amount of naturally occurring data is insufficient to explore the effects of the relative contributions due to multiple pure error sources (in particular each a,/ai
Lin and Siewiorek REF found that error logs usually consist of more than one failure process, making it imperative to collect these logs over extended periods of time.
Error Log Analysis: Statistical Modeling and Heuristic Trend Analysis
IEEE Transactions on Reliability
Computer Science