Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation (2020)

Abstract: Many recent Short Answer Scoring (SAS) systems have employed Quadratic Weighted Kappa (QWK) as the evaluation measure of their systems. However, we hypothesize that QWK is unsatisfactory for the evaluation of the SAS systems when we consider measuring their effectiveness in actual usage. We introduce a new task formulation of SAS that matches the actual usage. In our formulation, the SAS systems should extract as many scoring predictions that are not critical scoring errors (CSEs). We conduct the experiments in our new task formulation and demonstrate that a typical SAS system can predict scores with zero CSE for approximately 50% of test data at maximum by filtering out low-reliablility predictions on the basis of a certain confidence estimation. This result directly indicates the possibility of reducing half the scoring cost of human raters, which is more preferable for the evaluation of SAS systems.

## introduction the automated short answer scoring (sas) is a task of estimating a score of a short-text answer written as a response to a given prompt on the basis of whether the answer satisfies the rubrics prepared by a human in advance. sas systems have mainly been developed to markedly reduce the scoring cost of human raters. moreover, the sas systems play a central role in providing stable and sustainable scoring in a repeated and large-scale examination and (online) self-study learning support system @xcite @xcite . the development of the sas systems has a long history @xcite . many recent previous studies, e.g., @xcite @xcite , utilize quadratic weighted kappa (qwk) @xcite ) as a measure for the achievement and for the comparison of the performances of the sas systems. qwk is indeed useful for measuring and comparing the overall performance of each system and the daily developments of their scoring models. in our experiments, however, we reveal that the sas systems with high qwk potentially incur serious scoring errors (see experiments in section 5.3). such serious scoring errors are rarely incurred by trained human raters, therefore, we need to avoid containing this type of errors to ensure the sufficient scoring quality, for use in the scoring of commercial examinations, of sas systems. when we strictly focus on measuring the effectiveness of the sas systems in actual usage, qwk seems unsatisfactory for the evaluation of the sas systems. here, we assume that the following procedure is a realistic configuration for utilizing the sas systems in actual usage: (1) apply a sas system to score each answer, (2) treat the predicted score as the final decision if the predicted score is highly reliable, proceed to the next step otherwise, and (3) discard the unreliable predicted score and reevaluate the answer by a human rater as the final decision. therefore, we aim to establish an appropriate evaluation scheme for accurately estimating the effectiveness of the sas systems in actual usage instead of the current de facto standard evaluation measure, qwk. to do so, we first introduce a key concept critical scoring error (cse), which reflects unacceptable prediction error. specifically, cse refers to the observation that the gap between a predicted score and the ground truth is larger than a predefined threshold, which, for example, can be determined by an average gap in human raters. then, in our task formulation, the goal of the automated sas is to obtain as many predictions without cse as possible, which directly reflects the effectiveness of the sas models in the actual usage. we also in- student response conflicts of interest in western culture are formed on the basis of god vs. human, … score: 4 points prompt explain what the author means by the phrase "this tension has caused several different philosophical viewpoints in western culture" figure 1 : example of a prompt and a student's shorttext response excerpted from the dataset proposed by @xcite . the allotment score of this prompt is 16, and this response is assigned four points by a human rater. note that the prompt and the response are translated from the original ones given in japanese. in our experiments, we select two methods, i.e., posterior probability and trust score @xcite , as case studies of estimating whether or not each prediction is reliable. we use those two confidence estimation methods to obtain a set of highly reliable predictions. the experimental results show that the sas systems can predict scores with zero cse for approximately 50% of test data at maximum by filtering low-reliability predictions. as an example, in figure 1 , for a short answer question, a student writes a short text as a response to a given prompt. a human rater marks the response on the basis of the rubrics for the prompt. similarly, given a student response @xmath0, x 2 , ..., x n } for a prompt allotted n points, our short answer scoring task can be defined as predicting a score of @xmath1, ..., n } for that response. sas models are often evaluated in terms of the agreement between the scores of a model prediction and human annotation with qwk. qwk is calculated as: n is allotment score defined for a prompt. ## filtering out low-reliability estimation using confidence score as described in equation 8, the quality of the confidence measure is important for our task configuration. in this paper, we employ two methods for computing the confidence score: (1) posterior probability of the classification model and (2) trust score @xcite as case studies. ## conclusion and future work in this paper, we introduced a new formulation of the sas task to evaluate the effectiveness of the sas systems in actual usage. we defined the concept of a critical scoring error (cse), which represents unacceptable prediction errors. then, we formulate the objective of the task to obtain as many predictions without cse as possible. the experimental results show that by using our proposed procedure of selecting reliable predictions, sas systems can predict scores with zero cse for approximately 50% of test data at maximum. this result directly indicates the possibility of reducing half scoring cost of human raters, which, we believe, is highly preferable for the evaluation of sas systems. our study revealed some potential for a better task formulation of sas that links to actual usage. however, some issues remain, for example, how to determine the effective threshold τ that can strictly guarantee zero cse is still unknown. this is one major challenge regarding our formulation. moreover, we must develop a method for more accurately estimating the confidence scores, which is our primary focus in the next step.