Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability (2021)

Abstract: When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012). Metrics such as Krippendorff’s alpha or Cohen’s kappa are typically required to be above a threshold of 0.6 (Landis and Koch, 1977). These absolute thresholds are unreasonable for crowdsourced data from annotators with high cultural and training variances, especially on subjective topics. We present a new alternative to interpreting IRR that is more empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen’s (1960) kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.

## introduction much content analysis and linguistics research is based on data generated by human beings (henceforth, annotators or raters) asked to make some kind of judgment. these judgments involve systematic interpretation of textual, visual, or audible matter (e.g. newspaper articles, television programs, advertisements, public speeches, and other multimodal data). when relying on human observers, researchers must worry about the quality of the data -specifically, their reliability @xcite . are the annotations collected reproducible, or are they the result of human idiosyncrasies? respectable scholarly journals typically require reporting quantitative evidence for the inter-rater reliability (irr) of the data @xcite . cohen's kappa @xcite or @xcite ) is expected to be @xcite above a certain threshold to be worthy of publication, typically 0.6 (landis and @xcite . similar irr requirements for human annotations data have been followed across many fields. in this paper we refer to this absolute interpretation of irr as the landis-koch approach (fig. 1 ). this approach has been foundational in guiding the development of widely used and shared datasets and resources. meanwhile, the landscape of human annotations collection has witnessed a tectonic shift in recent years. driven by the datahungry success of machine learning @xcite , there has been an explosive growth in the use of crowdsourcing for building datasets and benchmarks @xcite . we identify three paradigm shifts in the scope of and methodologies for data collection that make the landis-koch approach not as useful in today's settings. a rise in annotator diversity in the precrowdsourcing era lab settings, data were typically annotated by two graduate students following detailed guidelines and working with balanced corpora. over the past two decades, however, the bulk of data are annotated by crowd workers with high cultural and training variances. a rise in task diversity there has been an increasing amount of subjective tasks with genuine ambiguity: judging toxicity of online discussions @xcite , in which the irr values range between 0.2 and 0.4; judging emotions expressed by faces @xcite , in which more than 80% of the irr values are below 0.6; and a/b testing of user satisfaction or preference evaluations @xcite , where irr values are typically between 0.3 and 0.5. a rise in imbalanced datasets datasets are no longer balanced intentionally. many high-stakes human judgements concern rare events with substantial tail risks: event security, disease diagnostics, financial fraud, etc. in all of these cases, a single rare event can be the source of considerable cost. high class imbalance has led to many complaints of irr interpretability @xcite @xcite . each of these changes individually has a profound impact on data reliability. together, they have caused a shift from data-from-the-lab to datafrom-the-wild, for which the landis-koch approach to interpreting irr is admittedly too rigid and too stringent. meanwhile, we have seen a drop in the reliance on reliability. machine learning, crowdsourcing, and data research papers and tracks have abandoned the use and reporting of irr for human labeled data, despite calls for it @xcite . the most cited recent datasets and benchmarks used by the community such as squad @xcite , imagenet @xcite , freebase @xcite , have never reported irr values. this would have been unthinkable twenty years ago. more importantly, this is happening against the backdrop of a reproducibility crisis in artificial intelligence @xcite . with the decline of the usage of irr, we have seen a rise of ad hoc, misguided quality metrics that took its place, including 1) agreement-%, 2) accuracy relative to consensus, 3) accuracy relative to "ground truth." this is dangerous, as irr is still our best bet for ensuring data reliability. how can we ensure its continued importance in this new era of data collection? this paper is an attempt to address this problem by proposing an empirical alternative to interpreting irr. instead of relying on an absolute scale, we benchmark an experiment's irr against two baseline measures, to be found in a replication. replication here is defined as re-annotating the same set of items with a slight change in the experimental setup, e.g., annotator population, annotation guidelines, etc. by fixing the underlying corpus, we can ensure the baseline measures are sensitive to the experiment on hand. the first baseline measure is the annotator reliability in the replication. the second measure is the annotator reliability between the replications. in section 3, we present a novel way of measuring this. we call it cross-kappa (κ x ). it is an extension of @xcite kappa and is designed to measure annotator agreement between two replications in a chance-corrected manner. we present in appendix a the international replication (irep) dataset, foot_0 a large-scale crowdsourced dataset of four million judgements of human facial expressions in videos. the dataset consists of three replications in mexico city, budapest, and kuala lumpur. 2 our analysis in section 4 shows this empirical approach enables meaningful interpretation of irr. in section 5, we argue xrr is a sensible way of measuring the goodness of crowdsourced datasets, where high reliability is unattainable. while we only illustrate comparing annotator populations in this paper, the methodology behind the xrr framework is general and can apply to similarly replicated datasets, e.g., via change of annotation guidelines. ## related work to position our research, we present a brief summary of the literature in two areas: metrics for measuring annotator agreement and their shortcomings (section 2.1), comparing replications of an experiment (section 2.2). ## cross-replication reliability (xrr) data reliability can be assessed when a set of items are annotated multiple times. when this is done by a single rater, intra-rater reliability assesses a person's agreement with oneself. when this is done by two or more raters, inter-rater reliability (irr) assesses the agreement between raters in an experiment. we propose to extend irr to measure a similar notion of rater-rater agreement, but where the raters are taken from two different experiments. we call it cross-replication reliability (xrr). these replications can be a result of re-labeling the same items with a different rater pool, annotation template, or on a different platform, etc. we begin with a general definition of @xcite kappa. we extend it to cross-kappa (κ x ) to measure cross-replication reliability. we then use this foundation to define normalized κ x to measure similarity between two replications. ## applying xrr to the irep dataset as a standalone measure, irr captures the reliability of an experiment by encapsulating many of its facets: class imbalance, item difficulty, guideline clarity, rater qualification, task ambiguity, etc. as such, it is difficult to compare the irr of different experiments, or to interpret their individual values, because irr is tangled with all the aforementioned design parameters. for example, we cannot attribute a low irr to rater qualification without first isolating other design parameters. this is the problem we try to solve with xrr by contextualizing irr with meaningful baselines via a replication. we will demonstrate this by applying this technique to the irep dataset (appendix a). we focus on a subset of 5 emotions for illustration purposes, with the rest of the reliability values provided in appendix b. in our analysis, irr is measured with @xcite kappa and xrr with κ x . we will refer to them interchangeably. ## measuring the quality of a crowdsourced dataset the irep dataset is replicated and is conducive to xrr analysis. however, in practice most datasets are not replicated. is xrr still useful? we present a specific use case of xrr in this section and argue that it is worth replicating a crowdsourced dataset in order to evaluate its quality. ## discussion there has been a tectonic shift in the scope of and methodologies for annotations data collection due to the rise of crowdsourcing and machine learning. in many of these tasks, a high reliability is often difficult to attain, even under favorable circumstances. the rigid landis-koch scale has resulted in a decrease in the usage and reporting of irr in most widely used datasets and benchmarks. instead of abandoning irr, we should adapt it to new ways of measuring data quality. the xrr framework presents a first-principled way of doing so. it is a more empirical approach that utilizes a replication as a reference point. it is based on two metricsκ x for measuring cross-replication rater agreement -and normalized κ x for measuring replication similarity. we opensource a large-scale replication dataset of facial expression judgements analyzed with the proposed framework. we show this framework can be used to guide our crowdsourcing data collection efforts. this is the beginning of a long line of inquiry. we outline future work and limitations below: confidence intervals for κ x confidence intervals for κ x and normalized κ x are required for hypothesis testing. though one can use the blockbootstrap for an empirical estimate, large sample behavior of these metrics needs to be studied.