Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations (2022)

Abstract: It is commonly claimed that inter-annotator agreement (IAA) is the ceiling of machine learning (ML) performance, i.e., that the agreement between an ML system’s predictions and an annotator can not be higher than the agreement between two annotators. Although Boguslav & Cohen (2017) showed that this claim is falsified by many real-world ML systems, the claim has persisted. As a complement to this real-world evidence, we conducted a comprehensive set of simulations, and show that an ML model can beat IAA even if (and especially if) annotators are noisy and differ in their underlying classification functions, as long as the ML model is reasonably well-specified. Although the latter condition has long been elusive, leading ML models to underperform IAA, we anticipate that this condition will be increasingly met in the era of big data and deep learning. Our work has implications for (1) maximizing the value of machine learning, (2) adherence to ethical standards in computing, and (3) economical use of annotated resources, which is paramount in settings where annotation is especially expensive, like biomedical natural language processing.

## introduction it is standard when conducting machine learning (ml) and natural language processing (nlp) work to calculate inter-annotator agreement (iaa) metrics like cohen's kappa @xcite . this is done not just for annotation quality control, but also as a comparison for machine learning models' performance. in particular, it has commonly been claimed -by some of the most prominent researchers in ml and nlp @xcite -that iaa places an upper bound or ceiling on the performance of machine learning models. when researchers claim this and their model reaches iaa, they are implicitly suggesting (or at least it follows) that the model has performed as well as possible or has solved the task for that dataset, and/or that the dataset cannot be used to drive further development of ml models. despite the prominence of this claim, however, @xcite reported that neither they nor a professional literature search service could find evidence in support of it. this is concerning for at least two reasons. first, as @xcite say, "if the assumption [that iaa bounds ml] turns out not to be supported. . . we may be mis-estimating the actual performance of our @xcite systems. in particular, we may be over-estimating the quality of their performance by under-estimating how good [performance] could potentially be" (pg 298). this underestimation of the maximum possible performance may lead to the development of poorer models under the belief that they have achieved maximum capacity. moreover, as noted by @xcite , such misestimation may violate ethical standards concerning accurate characterization of the limitations of computer systems (e.g., acm code of ethics and professional conduct 2.7 @xcite ; see also @xcite recommendations for safe, effective use of clinical decision support systems). second, and relatedly, if a modeler stops using an annotated dataset to drive ml development once ml performance on that dataset reaches iaa, they may be underutilizing those annotations. this could be an enormous waste of money, since annotation is often one of the most expensive components of an ml/nlp project, especially in biomedical nlp where the time of annotators (often biomedical experts) is especially expensive. for example, @xcite noted that then stateof-the-art word embeddings had reached iaa on existing word relatedness benchmark datasets (e.g., @xcite . believing that iaa was the upper bound of ml, they therefore believed that such datasets could no longer be used to drive development of different word embedding models. this led them to collect new annotations of word similarity, yielding the benchmark simlex-999. their abstract lays out this logic: "further, unlike existing gold standard evaluations, for which automatic approaches have reached. . . the interannotator agreement ceiling, state-of-theart models perform well below this ceiling on simlex-999. there is therefore plenty of scope for simlex-999 to quantify future improvements to distributional semantic models, guiding the development of the next generation of representation-learning architectures." (pg. 1) if iaa does not in fact bound ml, then the older word relatedness benchmarks could have actually been used to "guide. . . the development of the next generation of representation-learning architectures", and there would have been less need to spend time and money annotating simlex-999. given that @xcite has been cited over 1000 times according to google scholar, other researchers may have absorbed and replicated their logic, which would be concerning if the claim is not really true. indeed, a stack exchange post (tomas , https://stats.stackexchange.com/users/84364/tomas ) roughly contemporaneous with @xcite suggests that this logic may be widespread. despite the popularity and stakes of the claim that iaa bounds ml, @xcite found, across 6 papers, 20 ml systems that outperform iaa, on tasks ranging from entity recognition in clinical notes @xcite , to deception detection @xcite (and see @xcite for earlier evidence that ml can outperform iaa in information retrieval foot_0 ). however, claims that iaa bounds ml performance have persisted. this is seen in both biomedical and broader ml/nlp, in (1) papers that are often cited much more than @xcite and published in high impact outlets including jama network, amia, nature human behavior, and acl (e.g., @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , (2) machine learning lectures at well-known universities including university of pittsburgh @xcite , university of edinburgh @xcite , and city university of new york (cuny, @xcite and in slides by noted nlp textbook authors jurafsky and martin @xcite , and (3) online posts and social media discussions by prominent machine learning users (e.g., @xcite . it is not entirely clear why the claim that iaa bounds ml survived boguslav and cohen's counterexamples, but we suspect at least two factors are at play. first, @xcite was published in a fairly specialized journal (studies in health technology and informatics), and therefore may not have reached as many ml/nlp practitioners as it could or should have. consistent with this, as of @xcite has been cited only 7 times, according to google scholar. second, we suspect that the issue deserves a broad proof of concept based on simulations, in addition to the empirical examples raised by boguslav and cohen. simulations would be complementary to the real-world evidence brought by boguslav and cohen, in at least two ways. first, simulations allow us to simplify the problem to its essence, which may be clarifying in ways that realworld studies, with all their potentially distracting idiosyncrasies, are not. second, simulations allow us to precisely control and test different potentially relevant annotation and modeling factors, and therefore better understand when/how/why a model can or can't beat iaa. therefore, the aim of this study is to use a comprehensive set of computational simulations to bolster the evidence that iaa is not the upper bound on ml performance. the rest of this paper is organized as follows. we start with simplified simulations that capture the basic elements of an ml pipeline with two annotators and train a supervised model with these annotations (experiment 1). we then relax various assumptions of this setup to simulate potentially more realistic settings, and to better understand the conditions under which an ml model can outperform iaa (experiment 2). in both experiments, the general approach is to (a) simulate two annotators' annotations on a test set (where both annotators label all samples of this set); (b) simulate their annotations on disjoint halves of a training set; (c) train an ml model on the training set annotations; and ## experiment 2 the conditions of experiment 1 are intentionally oversimplified from real-world conditions. to better understand the range of conditions under which ml can or cannot outperform iaa, we next introduce some additional complexity in the simulations. ## general discussion in a comprehensive range of simulations, we showed that, contrary to popular belief @xcite , inter-annotator agreement is not the upper bound on machine learning performance. we showed this is the case even if (and especially if) annotators are noisy and differ in their underlying classification functions, as long as the ml model is reasonably well-specified. while we think noisy annotators with (possibly small) systematic individual differences are the norm rather than the exception, well-specified models have been elusive for a long time in domains with unstructured data like (biomedical) nlp or machine learning. this was especially true in decades past, when the belief that iaa bounded ml proliferated, and this illspecification likely led ml models to underperform iaa. however, reasonably well-specified models are likely to be increasingly attainable in today's era of big data, increased computing power, and correspondingly complex nonlinear models like deep neural networks. although these real-world cases involve much more complex data and models than we simulated here, we believe our conclusions still apply, and we therefore expect to see more empirical cases of ml outperforming iaa (like those in @xcite . likewise, although we focused on binary classification here, we expect our results to generalize straightforwardly to other settings, like multiclass classification or regression. on the other hand, whether and how much a model will beat iaa depends, as we have shown, on the degree of model misspecification, the degree of noise in annotators' judgments, the degree of individual differences in the annotators, and possibly other factors. an ml practitioner might therefore wish to determine, given a particular annotated dataset, how well-specified the model must be in order to beat iaa by a given margin. modeling one's annotators (e.g., passonneau and carpenter, 2014), and their noise levels and individual differences, may be useful here. beyond this, it is unclear how best to perform such an analysis, and thus we leave this to future work. for the time being, then, we simply recommend that researchers not claim that iaa is the ceiling of ml performance on their dataset. (relatedly, for consideration of what, if not iaa, constitutes the upper bound on ml performance, we refer the reader to the discussion section of @xcite . we realize that the simulations are so simple that our results and their implications may seem obvious. to an extent, we share this impression. at the same time, the persistence of the belief that iaa bounds ml performance, despite any evidence or argument in support of this claim, and despite empirical evidence contrary to the claim @xcite , suggests that the results are not intuitive, at least for a large number of practicing ml users (the smaller number of theoretical statistics and machine learning researchers may not be surprised by the present results). we are not entirely certain why the belief that iaa bounds ml has persisted -and, to some extent, this is a psychological and sociological question outside the scope of our work -but we suspect there are at least a few culprits. first, as @xcite pointed out, this belief makes our models appear better than they are, and it may be the case that ml users were therefore eager to believe that iaa bounded ml. second, as noted above, most models to-date have been (enormously) misspeci-fied, so most models will tend to fall short of iaa. third, as we showed in experiment 1, iaa positively correlates with ml performance. these latter two facts combined may give the appearance of iaa "pushing down on" ml performance (see especially @xcite and their figure 1 or @xcite and their figure 3 for possible cases of this reasoning). regardless of the reasons that the belief iaa bounds ml persisted in the past, our results ought to help dispel this belief in the future, and thereby help researchers realize the full potential of machine learning models, adhere to ethical standards in reporting the performance of computational systems, and use expensive annotated resources more efficiently.