Revisiting Active Learning under (Human) Label Variation (2025)

Abstract: Access to high-quality labeled data remains a limiting factor in applied supervised learning. Active learning (AL), a popular approach to optimizing the use of limited annotation budgets in training ML models, often relies on at least one of several simplifying assumptions, which rarely hold in practice when acknowledging human label variation (HLV). Label variation (LV), i.e., differing labels for the same instance, is common, especially in natural language processing. Yet annotation frameworks often still rest on the assumption of a single ground truth, overlooking HLV, i.e., the occurrence of plausible differences in annotations, as an informative signal. In this paper, we examine foundational assumptions about truth and label nature, highlighting the need to decompose observed LV into signal (e.g., HLV) and noise (e.g., annotation error). We survey how the AL and (H)LV communities have addressed—or neglected—these distinctions and propose a conceptual framework for incorporating HLV throughout the AL loop, including instance selection, annotator choice, and label representation. We further discuss the integration of large language models (LLM) as annotators. Our work aims to lay a conceptual foundation for (H)LV-aware active learning, better reflecting the complexities of real-world annotation.

## introduction prediction algorithms play a central role in many natural language processing (nlp) tasks, like hate speech detection @xcite , sentiment analysis @xcite , or natural language inference (nli; @xcite . for training such supervised machine learning (ml) models, a notable amount of labeled training data is necessary. however, acquiring high-quality labels is expensive as human crowd workers or, even more expensive, domain experts need to annotate the data. a popular scheme to efficiently guide the annotation process and allocate annotation budgets is active learning (al; @xcite . al aims to maximize the expected predictive performance of the resulting model while minimizing the required number of annotations; often done by iterating the following three steps: (1) training the ml model on available labeled data. (2) selecting new instances for labeling from a pool of unlabeled data, usually based on an acquisition function. (3) labeling these with an oracle. those steps, which are repeated until the available annotation budget is depleted or the model has reached its target accuracy, rest on the following assumptions: a1 there exists a single ground truth label per instance. ## assumptions about truth in annotation when observing lv in human annotations, it is important to recognize that this variation may arise from both error and hlv @xcite , which can be present simultaneously. throughout this work, we use lv to refer to the observed differences in annotation, which can be decomposed into signal, such as hlv, and noise, such as actual annotation error. reflecting on the underlying assumptions about the true labels is crucial, as it helps to distinguish between these sources of lv, or, in other words, aids the "interpretation of any observed annotator disagreement" @xcite . ## views on label variation and active learning in what follows, we outline key stages in how different fields have approached label variation, a phenomenon discussed under various terminologies and theoretical perspectives, and illustrate them with literature examples. starting from work documenting its occurrence across diverse tasks, we move from approaches that neglect or seek to mitigate lv, to studies that measure variation mainly to steer away from high lv instances. subsequently, we summarize recent perspectives that embrace (h)lv as a valuable signal, integrating it into learning objectives through distributional labels and motivating its decomposition. we then examine how the field of al has responded to, incorporated, or overlooked these diverse understandings of lv in its methodological developments. ## the active learning loop revisited in the following, we discuss the consequences of the assumptions about truth in annotation and the nature of the labels ( §2) on each of the steps of the al loop (as visualized in figure 1 ). ## conclusion in this work, we provide an overview of the crucial connection between the fields of (human) label variation and active learning. our comprehensive overview of the existing literature in the individual fields helps building bridges between different, but connected, streamlines of research, paving the way for the identification of critical aspects to consider in the al loop in the presence of hlv. our critical assessment of these aspects aims to further point out potential avenues for future research to deal with them in a more nuanced and reflective manner. in doing so, we uncover several crucial assumptions about labels which are often implicitly made in traditional al. however, we argue that they need to be made explicit. while providing a unified and implemented solution to the discussed problems is beyond the scope of the paper, we still hope to contribute to ongoing research debates on (h)lv by providing a fresh perspective from a different angle on existing problems and encourage new work addressing label-variation-aware active learning.