Measuring Label Ambiguity in Subjective Tasks using Predictive Uncertainty Estimation (2025)

Abstract: Human annotations in natural language corpora vary due to differing human perspectives. This is especially prevalent in subjective tasks. In these datasets, certain data samples are more prone to label variation and can be indicated as ambiguous samples.

## introduction natural language processing often relies on annotated corpora. due to the subjective nature of language @xcite , annotation tasks often involve subjective judgments, where the meaning of text can be open to multiple interpretations due to personal perceptions, cultural backgrounds or figure 1 : example text snippet for emotion classification, showing the diverse emotion labels assigned by a group of annotators. given these labels, we calculate the empirical probability distribution over classes. we use the characteristics of this distribution to define the label ambiguity score for the given text snippet. some annotators might label this as positive due to its praise of originality, while others might perceive it as negative if they prefer traditional narratives. such discrepancies highlight the difficulty in assigning definitive labels to subjective content @xcite . current models excel in well-defined tasks with clear, objective labels, such as spam detection, where the distinction between spam and not-spam is relatively straightforward. however, they often underperform in subjective tasks due to their inability to account for label ambiguity @xcite . these models tend to pro-vide overconfident predictions even on inherently ambiguous samples, lacking mechanisms to reflect uncertainty in their outputs @xcite . this overconfidence can lead to misguided trust in the model's predictions and obscure the identification of samples, i.e. annotatable items, that require further human review or special attention @xcite . furthermore, traditional evaluation metrics and training methodologies do not address the challenges posed by label ambiguity sufficiently @xcite . models are usually trained to minimize error, based on the assumption that there is a single correct label for each sample, which is not always the case in subjective tasks @xcite . this can result in models that are ill-equipped to handle the variability present in real-world data @xcite . the core problem addressed in this paper is the lack of effective methodologies for detecting and quantifying label ambiguity in text classification models. without proper identification and handling of ambiguous samples, models cannot differentiate between confidently correct predictions and those that are uncertain due to inherent ambiguity in the data. this limitation may hinder the development of reliable nlp systems capable of managing the complexities of human language interpretation, particularly in applications where understanding nuance and subjectivity is crucial. to address this problem, the paper investigates whether techniques for estimating uncertainty in model predictions can serve as a means to measure label ambiguity. label ambiguity is often demonstrated in datasets with crowd-sourced annotations, which exhibiti varying degrees of annotator agreement. for instance, in the goemotions dataset -a corpus for fine-grained emotion classification @xcite -some text samples receive unanimous labels, while others have annotations spread across multiple emotion categories. the variance in annotations indicates the level of ambiguity for each sample. traditional models might still assign high confidence to a single label, disregarding the underlying uncertainty reflected in the annotators' disagreement @xcite . given many annotators for each sample, we frame the empirical probability distribution over classes as a ground truth measure for sample-level ambiguity, as shown in figure 1 . this allows us to evaluate how well the sample-level uncertainty scores from various techniques align with ambiguity, by comparing them against the empirical probability distribution. in an additional ambiguity detection experiment, we define a threshold and have the models, equipped with stated uncertainty estimation techniques, predict which samples are ambiguous; samples with uncertainty scores within the threshold are marked as ambiguous. our contributions can be summarized as follows: ## evaluation data for label ambiguity in this section, we outline the evaluation data and metrics employed to investigate label ambiguity in subjective tasks. we utilize publicly available datasets with inherent annotation ambiguity, each annotated with multi-annotator labels, described in section 2.1. we define the label ambiguity score as the entropy of the empirical probability distribution over annotator labels, explained in section 2.2. ## methods we describe our methodology for uncertainty estimation to assess label ambiguity. our goal is to use uncertainty estimation either to directly predict the label ambiguity score or to approximate the full label distribution across classes. we detail the uncertainty estimation techniques employed in section 3.2, and explain how we derive an uncertainty score from the model outputs in section 3.3. ## experiment: measuring label ambiguity in the first experiment, we evaluate the effectiveness of the uncertainty estimation techniques in measuring label ambiguity. here, we compare how correlated the ambiguity and uncertainty scores are, as well as how close the empirical and predicted distributions are. ## experiment: detecting ambiguous samples this experiment demonstrates our methodology for detecting ambiguous samples in text classification using model uncertainty estimates. we apply percentile-based thresholds and flag samples that exceed these thresholds. with this, we assess the overlap between model-identified and annotatoridentified ambiguous samples and evaluate how ## related work there have been numerous studies addressing human label variation and label ambiguity. @xcite highlighted the variability in annotations obtained from non-expert annotators and the impact of this variability on nlp tasks. they demonstrated that aggregating multiple annotations can improve the quality of labels. another study proposed leveraging annotator disagreement instead of resolving it, suggesting that disagreement can provide valuable information.they advocated for models that learn from soft labels reflecting annotator probabilities rather than hard labels @xcite . we include this as our oracle softmax approach. uncertainty estimation techniques have gained attention as a means to quantify model confidence @xcite . in the context of deep learning, methods such as monte carlo dropout @xcite approximate bayesian inference by performing dropout at inference time, enabling models to estimate predictive uncertainty. similarly, deep ensembles @xcite improve uncertainty estimation by training multiple models with different initializations and aggregat- malinin and gales (2018) introduced prior networks to model predictive uncertainty, distinguishing between data uncertainty and model uncertainty in text classification tasks. recent research has begun to explore the relationship between model uncertainty and label ambiguity. braiek and khomh (2024) studied how incorporating human-like uncertainty into models can improve robustness in image classification tasks. they showed that models trained with uncertain labels can better handle ambiguous inputs. despite these advancements, there is limited work specifically focusing on leveraging uncertainty estimation techniques to detect label ambiguity arising from annotator disagreement in subjective text classification. ## conclusion in this paper, we focused on three subjective tasks of great interest: sentiment, emotion, and hate speech classification. for each task, we used public datasets with published multi-annotator labels. for every sample in these datasets, we defined a label ambiguity score as the entropy of the annotator label distribution, which measures the inherent randomness in the labeling process. we assessed the effectiveness of uncertainty estimation in quantifying label ambiguity. our evaluation included three techniques-deep ensemble, monte carlo dropout, and label smoothing-which we compared with both a baseline softmax model and an oracle softmax approach, the latter serving as an upper performance bound. for each method, we computed an uncertainty score defined as the entropy of the predicted label distribution. first, we evaluated whether predictive uncertainty techniques could effectively capture label ambiguity by calculating the correlation between uncertainty scores and label ambiguity scores. our findings indicate that these techniques-most notably deep ensembles-outperform the baseline softmax approach, with both deep ensembles and monte carlo dropout showing a low positive correlation with label ambiguity. additionally, we assessed the alignment between predicted class distributions and annotator class distributions. here, the label smoothing approach was successful in reducing the discrepancy between the distributions, making the predictions more ambiguity-aware. next, we applied the uncertainty estimation techniques to an ambiguity detection task, classifying each sample as either ambiguous or non-ambiguous using a fixed threshold. under these conditions, the deep ensemble approach achieved an error rate of about 40%, reducing it when compared to the baseline softmax approach. our results indicate that when fully leveraging annotator labels, as in the oracle softmax finetuning, the models' ability to quantify ambiguity improves, but the performance improvements remain modest. although the current uncertainty estimation techniques do not perfectly capture all aspects of label ambiguity, the findings are promising and indicate further research in this direction is needed. we believe this paper can provide a foundation for future research into more robust and effective methods for quantifying label ambiguity.