Comparing response latency and self-report methods for estimating levels of certainty in knowledge elicitation for rule-based expert systems
Many practitioners and theoreticians feel uncomfortable in using classic self-report, knowledge acquisition methods for assessing an expert's feelings of confidence or certainty about rules choices or decisions. This paper proposes and rests an alternative approach using response latency, or the time it rakes an expert to respond to a paired choice inquiry. The question is to learn if there is any merit in considering the response latency approach as a replacement for the classic method This investigation is concerned with using the analytic hierarchy process (AHP) to compare two methods for estimating a subject's feeling of certainty or confidence about their rule, choice or decision. One method is the older or classic method where a subject self-reports both the choice selected and the level of certainty associated with a choice-pair on the 1-to-9 scalp used with the ANF. The second is proposed by the authors and based on using only the choice and decision time or response latency. In the second case, the inverse of response latency is used to estimate a subject's degree of certainty or confidence in the selected choice in a paired comparison experiment. The response latency method has thr advantage of being unobtrusive, less prone to conscious censure, quicker to perform, requiring less effort, and possibly being less expensive to administer. Employing AHP methodology, priorities are used to represent the degree of confidence among choices or rules, and consistency ratios are used to estimate the degree of consistency detected by each method. To compare the two methods, 21 subjects were evaluated in a paired sample design. Employing a computer-administered questionnaire, subjects expressed relative levels of confidence for a set of potential recommendations and the time for each subject to respond was recorded unobtrusively. Using paired sample t tests, sufficient evidence was found to reject the null hypothesis (alpha = 0.025) that the mean of the difference in the priorities between the two methods was zero (p < 0.0001). In a second test, there was sufficient evidence to be at least 95% confident of accepting an alternative hypothesis that response latency produces a lower consistency ratio (higher consistency of the reports) than the self-report method (p < 0.0001). In addition to the tests of significance, the results were found to be practically significant and are discussed in the paper.
response latency certainty knowledge elicitation expert system knowledge acquisition decision making