Uncertain (Mis)Takes at L e W i D i-2025: Modeling Human Label Variation With Semantic Entropy (2025)

Abstract: The VariErrNLI task requires detecting the degree to which each Natural Language Inference (NLI) label is acceptable to a group of annotators. This paper presents an approach to VariErrNLI which incorporates measures of uncertainty, namely Semantic Entropy (SE), to model human label variation. Our method is based on the assumption that if two labels are plausible alternatives, then their explanations must be non-contradictory. We measure SE over Large Language Model (LLM)-generated explanations for a given NLI label, which represents the model uncertainty over the semantic space of possible explanations for that label. The system employs SE scores combined with an encoding of the inputs and generated explanations, and reaches a 0.31 Manhattan distance score on the test set, ranking joint first in the soft evaluation of VariErrNLI.

## introduction annotator disagreement has recently received more attention in nlp research @xcite @xcite . human label variation has consequences for the data, modeling, and evaluation in ml tasks @xcite . the question of data quality is related to distinguishing legitimate human label variation, which stems from different interpretations or opinions, from errors. in the context where a single label is correct, the problem of determining annotation reliability has been addressed by @xcite , who propose to evaluate the trustworthiness of each annotator in predicting the correct label. allowing human label variation adds a layer of difficulty to determining whether annotations are valid, since every combination of labels may be correct. some work has used the difference between annotator entropy and model entropy to predict which instances may require more annotations in an active learning setup @xcite . in this work we propose to solve the varierrnli task with uncertainty quantification (uq), specifically semantic entropy (se) @xcite . this approach expands on the work by @xcite by including the semantics of the input as well as sampled large language model (llm)generated explanations, and applies it to predicting the soft labels themselves rather than quantifying additional annotation needs. se has mostly been employed to detect hallucinations @xcite , where a prediction with a high se is interpreted as likely to have been hallucinated, given that the model is uncertain over the semantic space of the output. this is in line with prior work on uq, which focuses on model calibration @xcite and detecting noisy training data @xcite . @xcite propose to use uncertainty metrics such as similarity-sensitive entropy @xcite for detecting bias in machine translation, by leveraging the fact that uncertainty can also arise from ambiguity @xcite . models should be uncertain not only when they are not apt, but also when the input is ambiguous, where uncertainty is caused by more than one output being acceptable. ## task summary weber-genzel et al. (2024) introduced the natural language inference (nli)-inspired task vari-errnli, which contains both 1) valid annotator disagreement and 2) annotation errors. the dataset builds on the chaosnli @xcite ) dataset, which is composed of nli items with soft labels. a subset of chaosnli instances is annotated from scratch in two rounds by @xcite , with four annotators providing initial nli labels, and then returning to evaluate their own as well as their peers' judgments in a second round. annotations that are self-corrected are interpreted as errors and are not included in the gold label sets. varierrnli is one of the tasks in the lewidi shared task @xcite . in this paper we discuss a system that solves varierrnli with soft label prediction. that is, for an instance of varierrnli, we predict the acceptance rate of each label after the second round of annotation. this creates a multilabel binary classification setup with soft targets, where the score for each label reflects the proportion of annotators who accepted it. the example below illustrates an instance where after the second round of annotations, half the annotators believe that the entailment label is appropriate, three quarters of the annotators accept neutral as a valid label, and none support the contradiction label: context: "the next year, he built himself a palace, iolani, which can still be toured in honolulu." statement: "lolani was built in only 1 year." labels: entailment: 0.5, neutral: 0.75, contradiction: 0.0 in the shared task, systems are evaluated with soft labels, measuring how well the predicted label distribution matches the acceptability ratings of the different possible interpretations for each instance, as introduced by @xcite . specifically, manhattan distance is used to measure the difference between the predicted and target distributions, which has been shown to be particularly reliable for binary classification @xcite . the goal of our system is to be able to quantify the ambiguity in the nli instances. we postulate that if an instance is ambiguous, the explanations for different labels are likely to not entail one another. for instance, in the example from section 2, an explanation for the entailment label could read "the context states that he built himself a palace next year, which means that he finished it within the year", whereas the neutral label could be explained with "he may have started to build the palace the next year, but we do not know when he finished it". these explanations do not entail each other, which is indicative of ambiguity in the instance. in contrast, explanations for an instance which has only one valid interpretation should only have explanations which entail one another.