The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (2022)

Abstract: Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, thisconventional practice assumes that there exists a *ground truth*, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers.In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: *data, modeling and evaluation*. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the “problem” will lead to an open discussion on possible strategies to devise fundamentally new directions.

## introduction in natural language processing (nlp) much progress today is driven by fine-tuning large pretrained language models using an annotated dataset, assumed to be representative for a target language task of interest @xcite . this is analogously so in machine learning (ml) and computer vision (cv), where the target tasks differ, yet the conceptual pipeline remains the same: data, modeling, evaluation. despite the importance of annotated data-as it fuels all steps in this pipeline-a crucial assumption of today's learning systems is to rely on a single gold label per instance. the gold figure 1 : we propose the term human label variation to capture the fact that inherent disagreement in annotation can be due to genuine disagreement, subjectivity or simply because two (or more) views are plausible. the assumption of a ground truth (and taking the majority vote or the 'mode' of the human judgement distribution) makes sense when humans involved in labeling highly agree on the answer to the questions, such as "does this image contain a bird?", "is 'learn' a verb?", "what is the capital of italy?". however, this assumption often does not make sense-especially when language is involved. for example, on questions determining a word sense, questions such as "is this comment toxic?" or questions involving understanding indirect answers to polar questions like "q: hey. everything ok?" "a: i'm just mad at my agent" (see more examples in figure 2 ). while some disagreement is due to human labeling errors (cf. figure 1 arrow to the left and § 3), an increasing body of work has shown that irreconcilable variation between annotations is plausible and abundant @xcite @xcite (illustrated in figure 1 ). the observed variation can indeed be disagreement due to difficult cases, subjectivity or cases where multiple answers are plausible (cf. § 2). we argue that human label variation (hlv) provides rich information that should not be discarded. critically, to rely on a ground truth means we tacitly agree to continue: i) to create datasets that encode a single ground truth, ii) to develop models that are optimized towards a single preferred output, and iii) to evaluate models against a single ground truth. by continuing to do so, we might ask ourselves if we are climbing the right hill-or whether continuing to model a single ground truth hampers progress. in this position paper, we argue that neglecting variation in labeling is problematic, as it impacts all steps of the pipeline. traditionally, this variation has been considered a problem. we underline emerging works that instead believe this issue to be an opportunity. in fact, we believe it is essential to take human label variation into account for progress. human labels are bound to be scarce yet at the same time critical as they provide human interpretations and values. therefore, embracing it is necessary for human-facing nlp, i.e., technology which is by and for humans; inclusive and reliable. however, the research landscape is fragmented, and approaches often focus on either steps of the pipeline. therefore, in this paper we focus on the three core aspects of the pipeline: data, modeling and evaluation. in particular, i) we distill some of the on-going discussions in disparate (sub-)fields and propose a unified term; ii) we present and work out suggestions for each for the future; and iii) we provide a comprehensive repository of publiclyavailable data sets that allow studying human label variation, and invite the community to contribute. ## data and human label variation high-quality data is essential for any empirical scientific inquiry and has to satisfy the requirements of validity and reliability @xcite @xcite . however, for almost all tasks in nlp and cv irreconcilable disagreement between annotators has been observed @xcite . in light of this, the original definition of data reliability is questionable-it assumes labels follow a given standard. we might ask which standard? human annotations are needed to ground and make sense of language, images, speech etc. however, labelling data is difficult, particularly when dealing with an object of study as complex as language. take the illustration in figure 2 as example. while categories exist, their boundaries are fluid, or simply multiple options are plausible. ## modeling and human label variation there is a growing literature on methods on how to deal with hlv in learning. we categorize them into two camps: those that resolve variation, and those that embrace it. we will draw connections to surveys and the emerging literature, and discuss adoption of methods as well as gaps. the first big camp of research aims at resolving human label variation and includes: 1) aggregation and 2) filtering. it considers hlv as "problematic" or "noisy". consequently, a single (aggregated) label is obtained with presumably high agreement as the ground truth. aggregation is performed via majority voting or probabilistic aggregation methods, see @xcite for a survey and seminal works @xcite @xcite . aggregation is still the most widely-adopted solution for the problem today. however, aggregation by definition allows only one belief/label/category. this is very limiting, as often it is not just about disagreement or matter of subjectivity, but multiple options being plausible. filtering methods are advocated by some with the idea to remove data instances with low agreement @xcite @xcite . however, only using high-agreement instances can yield worse performance @xcite and it wastes data. the second camp of research instead aims at embracing human label variation. two broad directions include: 3) learning from un-aggregated labels (directly), or 4) enriching gold with human label variation. with regard to learning from unaggregated labels, methods of varying complexity exist, from model-agnostic methods such as repeated labeling @xcite used by e.g. de @xcite , to architecturespecific choices, e.g., adding a crowd layer (rodrigues and pereira, 2018), learning from soft labels @xcite and more; see the survey of @xcite . so far learning from un-aggregated labels directly has shown greater promise in classification tasks in cv than in nlp @xcite (evidence is scarce, see open issues). within nlp, a more studied direction is currently to enrich the gold label with human label variation, i.e., to learn from both the gold and the un-aggregated labels. methods in this category can be seen as part of the broader set of well open issues and our suggestions undoubtedly, there is increasing interest in studying methods to learn with human label variation (see figure 3 for our analysis of research papers). however, existing research is fragmented across (sub)-disciplines. we identify at least three diverse areas within nlp, with little to no overlap (as shown in learning from hlv heavily depends on data labeled with multiple annotators. in some settings, it might be difficult to obtain sizeable amounts of such data (however, as seen in section 2, more datasets are emerging). regarding learning, @xcite find that even small amounts of data can be helpful in a sequential fine-tuning setup, as also early work indicates @xcite ). an open challenge is to find the right balance between the amount of data collected and the number of annotators. overall, we hypothesize that the richness of information captured by human label variation has the potential to reduce data size requirements (possibly fewer instances but with more information captured in the human label distribution). it remains an open issue to connect with emerging works on learning with different amounts of annotation @xcite , which can also lead to novel architectures. a related important challenge is to tease apart errors from signal (e.g. @xcite . work on annotation error detection exists, cf. the very recent survey by @xcite or @xcite . it is though largely overlooked. this calls further for theoretical work on the notion of an what constitutes an error versus a hard case @xcite @xcite . this bears connections to emerging work in hci, in particular social computing @xcite , who look at the perception of system errors by humans, see also section 4, and earlier work in hci on crowdsourcing that allows for some errors @xcite . while embracing human label variation helps to regularize learning, the connection to a broader range of ml methods such as noise labeling or calibration remains highly relevant and a source of further inspiration @xcite @xcite . there are some initial studies that compare human disagreement with model confidence @xcite . overall, interest in calibration methods @xcite is increasing @xcite @xcite to counter overconfidence of neural classifiers @xcite . in contemporary work to this, we show that measuring calibration to human majority given inherent disagreements is theoretically and empirically problematic @xcite . as a first step, we propose instance-level measures of calibration that better capture the human label distribution. in future, it remains to be seen how to best use human label variation to make systems more trustworthy. finally, there is relevant interesting work that more deeply looks at data during learning. in nlp, recent seminal work by @xcite proposes data maps to investigate the behavior of a model on individual instances during training (training dynamics). they show that training a system on ambiguous instances identified via data maps helps to generalizes better in out-of-distribution evaluation @xcite . building on top of this work, @xcite show that the instances at the boundary of hard and ambiguous cases derived from small data maps aids active learning. this is further evidence that human uncertainty in labeling is beneficial for learning. it remains to be seen whether training dynamics can yield novel architectures for learning from hlv. ## evaluation and human label variation evaluation is of critical importance in empirical research fields such as ml, nlp and cv. it helps to choose one system over another, and to measure progress. however, current evaluation practices typically use accuracy against a gold standard. in many tasks this common practice is severely flawed. it obfuscates the truth about the state of ml models. it leaves a large gap between in-vitro and in-vivo evaluation. hci research has shown that metrics are not aligned with reality; audits of algorithms' performance have uncovered very poor results in practice, and that this disconnect is indicative of a larger disconnect on how ml and hci researchers evaluate their work @xcite . we believe this is an important take-away for nlp. we too often focus on single metrics, single components of the pipeline, in other words, on myopic in-vitro experimentation. open issues and our suggestions despite the increasing body of literature on methods for learning with hlv, a majority of the papers introducing new methods strikingly evaluate against hard labels (gold labels) (e.g. @xcite . if we want to take human label variation seriously, we need to shift our attention to evaluation that goes beyond hard labels (accuracy). as accuracy of all models can be high (at times), looking at only one metric (and, in fact a singleargmax-prediction) gives no indication on how reasonable a model is, yet alone how confident and trustworthy it is. research in ml, cv and nlp has started to incentivize hard and soft label evaluation. soft labels compare the human label distribution to model outputs. proposed soft metrics include: cross entropy, to capture how well the model captures humans' assessment not just of the top label, which is used in both cv @xcite and nlp @xcite ; entropy correlation proposed by @xcite , to compute pearson's correlation between instance-level entropy scores of human soft labels and model predictions; kullback-leibler divergence-based evaluation @xcite (either kl or jensen-shannon). others instead started to evaluate against individual annotators @xcite , measure f1 scores against data splits by different annotator agreement levels @xcite , data splits based on annotator clustering @xcite , data splits based on item difficulty based on entropy of the label distribution and semantic distance @xcite , and data splits based on annotator uncertainty flags @xcite . analogously as in section 3, it is an open issue to see whether tasks might need to have specific properties to be more suitable for one kind of evaluation over another. in general, we need better evaluation practices (besides soft and hard evaluation), particularly in light of the complexity of human label variation-and the reasons it arises, which might be due to uncertainty, background, task complexity, intra-coder reliability etc; see @xcite and in particular jiang and de marneffe (2022) for a discussion on disagreement sources; the latter recently developed a taxonomy for disagreement in natural language inference data. ## conclusions in this paper, we outline that human label variation impacts all steps of the traditional ml pipeline, and is an opportunity, not a problem. to move forward, we argue for a more comprehensive treatment of hlv, which considers all steps, to enable innovation: data, modeling and evaluation. to do so, and truly move beyond the current in-vitro setups, we need an open, interdisciplinary discussion. we hope to contribute to this discussion, and stipulate research with the released repository: https://github.com/ mainlp/awesome-human-label-variation . 1 limitations this position paper tries to be succinct while aiming at synthesizing a very broad notion-human label variation-that affects all steps dealing with learning from annotated data. therefore, this position paper is necessarily incomplete, as is the dataset repository that is provided. however, we hope that the repository and paper will lead to an open discussion and community uptake, as this is a big open issue and necessitates a broader, interdisciplinary treatment.