Using PRMSE to evaluate automated scoring systems in the presence of label noise (2020)

Abstract: The effect of noisy labels on the performance of NLP systems has been studied extensively for system training. In this paper, we focus on the effect that noisy labels have on system evaluation. Using automated scoring as an example, we demonstrate that the quality of human ratings used for system evaluation have a substantial impact on traditional performance metrics, making it impossible to compare system evaluations on labels with different quality. We propose that a new metric, PRMSE, developed within the educational measurement community, can help address this issue, and provide practical guidelines on using PRMSE.

## introduction nlp systems are usually trained and evaluated using human labels. for automated scoring systems, these would be scores assigned by human raters. however, human raters do not always agree on the scores they assign @xcite @xcite and the inter-rater agreement can vary substantially across prompts as well as across applications. for example, in the asap-aes data @xcite , the agreement varies from pearson's @xmath0.63 to @xmath1.85 across "essay sets" (writing prompts) . in many automated scoring studies, the data for training and evaluating the system are randomly sampled from the same dataset, which means that the quality of human labels may affect both system training and evaluation. notably, the effect of label quality on training and evaluation may not be the same. previous studies @xcite suggest that when annotation noise is relatively random, a system trained on noisier annotations may perform as well as a system trained on clean annotations. on the other hand, noise in the human labels used for evaluation can have a substantial effect on the estimates of system performance even if the noise is random. in this paper, our focus is the effect of noise in human labels on system evaluation. how do we compare two systems evaluated on datasets with different quality of human labels? while there exist several public data sets that can be used to benchmark and compare automated scoring systems, in many practical and research applications the scoring systems are customized for a particular task and, thus, cannot be evaluated appropriately on a public dataset. as a result, the research community has to rely on estimates of system performance to judge the effectiveness of the proposed approach. in an industry context, the decision to deploy a system is often contingent on system performance meeting certain thresholds which may even be codified as company-or industry-wide standards. a typical solution to the problem of different human-human agreement across evaluation datasets is to use human-human agreement itself as a baseline when evaluating a system @xcite . in this case, the system can be evaluated either via a binary distinction (did its performance reach human-human agreement?) or by looking at the differences in agreement metrics as measured between two humans and between a single human and the machine, known as "degradation" @xcite . yet how do we interpret these numbers? is a system that exceeds a humanhuman agreement of @xmath2.4 on one dataset better than another that performs just below a humanhuman agreement of @xmath3.9 on a different dataset? in this paper, we use simulated data to demonstrate that the rate of human-human agreement has a substantial effect on estimates of system performance, making it difficult to compare systems that are evaluated on different datasets. we also show that this problem cannot be resolved by simply looking at the difference between human-human and machine-human agreement. we then show that one possible solution is to use proportional reduction in mean squared error (prmse) @xcite , a metric developed in the educational measurement community, which relies on classical test theory and can adjust for human error when computing estimates of system performance. ## related work the effect of noisy labels on machine learning algorithms has been extensively studied in terms of their effect on system training in both general machine learning literature (see, for example, @xcite for a comprehensive review), nlp @xcite @xcite @xcite and automated scoring @xcite . one key insight that emerged from such work is that the nature of the noise is extremely important for the system performance. machine learning algorithms are greatly affected by systematic noise but are less sensitive to random noise @xcite . a typical case of random noise is when the labeling is done by multiple annotators which minimizes the individual bias introduced by any single annotator. for example, in a study on crowdsourcing nlp tasks, @xcite showed that a system trained on a set of non-expert annotations obtained from multiple annotators outperformed a system trained with labels from one expert, on average. the studies discussed so far vary the model training set, or training regime, or both while keeping the evaluation set constant. fewer studies have considered how inter-annotator agreement may affect system evaluation when the training set is held constant. these studies have shown that in the case of evaluation, the label quality is likely to have a substantial impact on the estimates of system performance even if the annotation noise is random. @xcite used simulated data to explore the effect of noisy labels on classifier performance. they showed that the performance of the model, measured using cohen's kappa, when evaluated against the 'real' (or gold-standard) labels was higher than the performance when evaluated against the 'observed' labels with added random noise. this is because for some instances, the classifier's predictions were correct, but the 'observed' labels contained errors. @xcite used two different datasets to train and evaluate an automated system for scoring spoken language proficiency. they showed that training an automated system on perfect labels did not give any advantage over training the system on noisier labels, confirming previous findings that automated scoring systems are likely to be robust to random noise in the data. at the same time, the choice of evaluation set led to very different estimates of system performance regardless of what data was used to train the system. metrics such as pearson's correlation or quadratically-weighted kappa, commonly used to evaluate automated scoring systems @xcite @xcite , compare automated scores to observed human scores without correcting for any errors in human scores. in order to account for differences in human-human agreement, these are then compared to the same metrics computed for the human raters using measures such as "degradation": the difference between human-human and humanmachine agreement @xcite . in this paper, we build on findings from the educational measurement community to explore an alternative approach where estimates of system performance are corrected for measurement error in the human labels. classical test theory @xcite ) assumes that the human holistic score is composed of the test's true score and some measurement error. a "true" score is defined as the expected score over an infinite number of independent administrations of the test. while such true scores are latent variables, unobservable in real life, their underlying distribution and measurement error can be estimated if a subset of responses is scored by two independently and randomly chosen raters. @xcite ; haberman et al. (2015); haberman and yao (2015); yao et al. (2019a,b); zhang et al. (2019) proposed a new metric called proportional reduction in mean squared error (prmse) which evaluates how well the machine scores predict the true score, after adjusting for the measurement error. the main contribution of this paper is a further demonstration of the utility of this metric in the context of automated scoring. outside of educational measurement, a similar approach has been been explored in pattern recognition by @xcite , for example, who used estimated error rates in human labels to adjust performance estimates. we further explore how agreement between human raters affects the evaluation of automated scoring systems. we focus on a specific case where the human rating process is organized in such a way that annotator bias is minimized. in other words, the label noise can be considered random. we also assume that the scores produced by an automated scoring system are on a continuous scale. this is typical for many automated scoring contexts including essay scoring (shermis, 2014), speech scoring @xcite and, to some extent, content scoring @xcite but, of course, not for all possible contexts: for example, some of the semeval 2013 shared tasks on short answer scoring @xcite use a different scoring approach. ## simulated data in this paper, we use simulated gold-standard (or "true") scores, human scores and system scores for a set of 10,000 responses. since "true" scores are not available for real data, using simulated data allows us to compare multiple raters and systems to the known ground-truth. 1 we focus on evaluation only and make no assumptions about the quality of the labels in the training set or any other aspects of system training. the only thing we know is that different human raters and different systems in our data set assign different scores and have different performances when evaluated against true scores. as our gold-standard, we use a set of continuous scores simulated for each response and consider these to be the correct "true" score for the response. note that the continuous nature of goldstandard scores allows us to capture the intuition that some responses fall between the ordinal score points usually assigned by human raters. to create such gold-standard scores, we randomly sampled 10,000 values from a normal distribution using the mean and standard deviation of human scores observed in a large-scale assessment (@xmath4.844, @xmath5.74). since the scores in the large-scale assessment we use as reference varied from 1 to 6, the gold-standard scores below 1 and above 6 were also truncated to 1 and 6 respectively. next, we simulated scores from 200 human raters for each of these 10,000 "responses". for each rater, its score for a response was modeled as the gold-standard score for the response plus a random error. we model different groups of raters: with low (inter-rater correlation @xmath6.4), moderate (@xmath7.55), average (@xmath8.65) and high (@xmath9.8) agreement. the correlations for different categories were informed by correlations we have observed in empirical data from various studies. the errors for each rater were drawn from a normal distribution with a mean of 0. we chose the standard deviation values used to sample the errors in order to create 4 categories of 50 raters, each defined by a specific average inter-rater correlation. since in most operational scenarios, human raters assign an integer score, all our simulated human scores were rounded to integers and truncated to lie in @xcite , if necessary. for each response, we also simulated 25 automated scores. like human scores, automated scores were simulated as gold-standard scores plus random error. we chose the standard deviation values used to sample the random errors so as to obtain specific levels of performance against the gold-standard scores: the worst system had a root mean squared error (rmse) of 0.74 score points while the best system had an error of 0.07 score points. since the interpretation of rmse depends on the score scale, we chose these values as the percentage of gold-standard score variance. to summarize, the final simulated dataset consisted of 10,000 "responses". each response had 1 "gold-standard" score, 200 "human" scores and 25 "system" scores. 3 4 problems with traditional metrics ## prmse with reference to true scores the simulations in the previous sections demonstrate that the values of metrics usually used to evaluate automated scoring systems are directly dependent on the quality of human ratings used to evaluate the system. in fact, the effect of human label quality can be so large such that two identical systems may appear drastically different while the performance of two very different systems may appear very similar. one possible solution is to collect additional ratings for each response from multiple raters as we showed in §4.3. this solution is likely to be too expensive to be feasible: for example, in our simulated dataset, we would need to collect at least 10 additional ratings for each re-sponse in order to obtain stable estimates of system performance, more if the rater agreement is low. the solution we propose comes from the educational measurement community and draws on test theory methods to adjust the system performance estimates for measurement error. ## discussion the performance of automated systems is often lower on data with lower human-human agreement. while this may mean that responses harder to score for humans are also harder to score for machines, our analyses show that this is not always true. furthermore, since subsets of the same dataset are often used for both system training and evaluation, separating the effect of noisy labels on training from that on evaluation may be impossible. in this paper, we showed that even for the same set of automated scores, estimates of system performance depend directly on the the quality of the human labels used to compute the agreement metrics. we also showed that using standard performance metrics to compare two systems may be misleading if the systems are evaluated against human scores with different inter-rater agreements. comparing system performance to human-human agreement using degradation does not resolve this issue. we proposed that a new metric, prmse, developed within the educational measurement community for evaluation is an effective way to obtain estimates of system performance that are adjusted for human-human agreement. prmse provides system evaluation against 'true' scores, thus making it possible to compare different systems on the same scale and offering a performance metric that is robust to the quality of human labels. we emphasize that prmse does not affect the evaluation results when the systems are evaluated on the same set of human labels, for example, in the context of a shared task or a benchmark dataset. however, it can help compare system performance across studies as well as within studies, for example, when the dataset includes multiple items with varying levels of human-human agreement in their respective human scores. the theory behind prmse makes certain assumptions about the nature of the rater error: it is assumed to be random with a mean of 0 and finite variance. furthermore, the rater error is assumed to be independent of the item and its true score. there are several steps one can take to make sure the data meets these assumptions. for example, a standard way to randomize rater error is to set up the scoring process in a way such that multiple raters each score a different set of responses. furthermore, one should additionally check whether human ratings have similar mean and variance. we note that other models discussed in the nlp literature (see §2), made other assumptions, for example that noisier labeling is more likely for some items ("hard" cases) than others. the performance of prmse under such conditions remains subject for future studies. finally, while prmse can adjust estimates of system performance for human error, it does not fully address the issue of different datasets. users of automated scoring still need to use their judgement -or additional extrinsic criteria -to decide whether two systems can be deemed comparable. ## practical guidelines for prmse we conclude with guidelines for using prmse. prmse values above 1 indicate that the doublescored sample is too small. • prmse should be used in combination with other metrics of human-machine agreement.