combined with removing the oldest one. This model updating procedure is expected to improve prediction accuracy and anomaly detection capability. [Insert Figure 3 here] V. Anomaly Detection Comparison across CE Models MAPE Comparison performing analytical procedures, auditors use various methods to make predictions on account numbers. It is desirable for expectation models to make forecasts as close to actual values as possible. Many prior AP studies evaluate expectation models in terms of prediction accuracy (Kinney 1978, Wild 1987, Dzeng 1994, Allen et al. 1999, Chen and Leitch 1998, Leitch and Chen 2003). Following this line of research, we compared the prediction accuracies for the CE models in this study using a measure of prediction accuracy called Mean Absolute Percentage Error (MAPE). Additionally, we compared the CE models on their error detection ability. MAPE, the absolute value of the difference between the predicted value and the actual value measured as a percentage of the actual value, is a commonly used metric of prediction accuracy. It is expected that a good model should have a small MAPE. The training set is first used to estimate each of the models. Then, each estimated model is used to make one-step-ahead forecasts and the forecast variance is calculated. After that, the model is updated based on the new data feeds in the hold-out set and the previous steps are repeated. Finally, all the absolute variances are summed up and divided by the total number of observations in the hold-out sample to compute the MAPE. The results for MAPE of Voucher predictions are presented in Table 2. [Insert here] The results indicate that as measured by the MAPE metric the prediction accuracies of models are close. The BVAR version of the VAR model has the best prediction accuracy (MAPE=0.3330), followed by the Subset VAR model (MAPE=0.3374), though the standard deviation for the BVAR model is slightly higher than that of the Subset VAR. The SEM has the lowest prediction accuracy of 0.3499. These prediction accuracies indicate that the forecasts generated by the expectation models usually differ from the reported amounts by approximately 30%.
when the AP model detects a seeded error in the hold-out sample, we examine the consequences on subsequent error detection if the seeded error is corrected by substitution of the original value before the model is used again. For comparison purpose, we test how our candidate CE models work with and without real time error correction. Unlike continuous auditing, anomalies are detected but usually not corrected immediately in traditional auditing. To simulate this scenario, we don’t correct any errors we seeded in the hold-out sample even if the AP model detects them. [Insert here] One can argue that it is possible in principle to utilize error correction in traditional auditing. However, since traditional auditors will have to investigate all the anomalies at the end of the audit period, the time pressure of the audit is likely to prevent them from rerunning their analytical procedures and conducting additional investigations over and over again to detect additional anomalies after some previously detected errors are corrected. Overall, the results show lower false negative error rates for all CE models with error correction, especially when the error magnitude is large (100% or more). This finding is consistent with the prior expectation. Under some rare circumstances, the non-correction model has slightly better detection rate than the correction model. All these exceptions occur when the error magnitude is small (no larger than 50 percent). Higher false positive error rates, mostly when error magnitudes are large, are observed for the error-correction models, which means that the error-correction models can detect more anomalies but at a cost of triggering more false alarms. However, a further investigation reveals that the false alarms are mostly caused by certain records in the holdout sample, which are called in this study “original anomalies”. These original anomalies are very likely to be caused by measurement errors since our dataset consists of un-audited operational data. This measurement error problem with non-audited data is also reported by previous studies (Kinney and Salamon 1982, Wheeler and Pany 1990). Because the non-correction model would not correct those undetected errors, the impact of original anomalies, whose values remain constant, would be eclipsed by the increase in seeded error magnitude. Therefore, the non-correction model would trigger fewer false alarms when the magnitude in seeded error increases. On the other hand, the impact of original anomalies would not decrease as the error-correction model would correct all detected errors.