From Disagreement to Understanding: The Case for Ambiguity Detection in NLI (2025)

Abstract: This position paper argues that annotation disagreement in Natural Language Inference (NLI) is not mere noise but often reflects meaningful variation, especially when triggered by ambiguity in the premise or hypothesis. While underspecified guidelines and annotator behavior contribute to variation, content-based ambiguity provides a process-independent signal of divergent human perspectives. We call for a shift toward ambiguity-aware NLI that first identifies ambiguous input pairs, classifies their types, and only then proceeds to inference. To support this shift, we present a framework that incorporates ambiguity detection and classification prior to inference. We also introduce a unified taxonomy that synthesizes existing taxonomies, illustrates key subtypes with examples, and motivates targeted detection methods that better align models with human interpretation. Although current resources lack datasets explicitly annotated for ambiguity and subtypes, this gap presents an opportunity: by developing new annotated resources and exploring unsupervised approaches to ambiguity detection, we enable more robust, explainable, and human-aligned NLI systems.

## introduction this paper takes a position on how disagreement in natural language inference (nli) is best understood and modeled. while prior work has often treated annotator disagreement as noisesomething to be minimized or resolved @xcite -we argue that such disagreement can reflect meaningful, coexisting interpretations grounded in linguistic ambiguity. nli, also known as recognizing textual entailment (rte) @xcite , aims to classify the relationship between a premise (p) and a hypothesis (h). suppose @xmath0, @xmath1, @xmath2, @xmath3. standard inference labels would assign entailment to (p1, h1), contradiction to (p1, h2), and neutral to (p2, h1). however, humans may diverge: (p1, h2) could also be neutral if likes is interpreted as distant admiration (e.g., of a celebrity). we adopt a perspectivist reframing of nli that treats variation in inference classifications not as a flaw but as an inherent feature of natural language understanding. we emphasize content-based ambiguity as a central source of disagreement and advocate for deeper exploration of its role in shaping inference judgments. we situate our analysis within a framework for handling ambiguous nli instances (figure 1 ). this framework first determines whether an input pair is ambiguous; if so, it disambiguates the pair into dis-tinct yet plausible human interpretations, enabling predictions aligned with each interpretation. nli is pivotal in understanding semantic relationships and is central to evaluating how well language models process natural language. nli benchmarks are typically constructed using humanannotated entailment labels @xcite . despite frequent disagreements among annotators on the "correct" label for a given premise-hypothesis pair, most nli research assumes a single "true" inference for each case. instances that deviate from this assumption are either filtered out during dataset construction @xcite or handled via majority vote @xcite , based on the belief that annotation disagreements reflect random error rather than systematic variation. however, this approach contradicts the original purpose of nli: to model what a reasonable, attentive, and informed human would plausibly infer from text @xcite . recent studies have challenged the assumption that annotation disagreements are mere noise, demonstrating instead that such disagreements exhibit reproducible patterns grounded in legitimate interpretive differences @xcite . this recognition has motivated efforts to model the full distribution of plausible human inferences @xcite . while these efforts are important, understanding why such differences arise is equally critical for developing systems that reflect multiple human interpretations. this position paper argues for an nli modeling goal that centers on identifying and categorizing ambiguity into recurring interpretive patterns, rather than merely modeling annotator distributions to capture coexisting human perspectives. we support this position through a review and analysis of existing research. the next section reviews work on modeling annotator label distributions and their limitations, motivating a closer look at sources of disagreement in nli. section 3 examines prior categorizations of these sources, section 4 highlights the unique role of ambiguity in premise-hypothesis pairs, and section 5 surveys current ambiguity-focused nli research and future directions. ## modeling annotator distribution most nli benchmarks are constructed using human-annotated entailment labels, which often result in cases where multiple annotators assign dif-ferent labels to the same premise-hypothesis pair. from the early days of nli research, scholars have expressed concerns about how to handle such disagreements @xcite . re-annotation of the rte1 development and training sets reveals substantial discrepancies between the original and new labels @xcite . even after filtering out problematic examples, human judges only achieve a 91% agreement rate. similar disagreements in the stanford natural language inference (snli) dataset complicate the process of learning robust decision boundaries for each entailment label @xcite . such cases are typically treated as "annotation noise," resolved by assigning a majority label under the assumption that one "true" inference exists for each premise-hypothesis pair. however, growing evidence suggests that these disagreements reflect systematic, reproducible variation rather than random error @xcite . in many cases, divergent annotations signal the existence of multiple plausible interpretations. current nli models, trained on majority-labeled benchmarks, struggle to capture the full distribution of human judgments and tend to perform better when annotator agreement is high @xcite . this highlights both a dependence on agreement and a failure to model collective human reasoning. @xcite further show that models trained on soft labels-distributions over annotator responses-better approximate human judgments and improve single-label prediction accuracy. these findings have inspired a growing line of research focused on modeling human opinion distributions. for example, the uncertain natural language inference (unli) framework @xcite proposes predicting subjective probabilities of entailment rather than coarse categorical labels. while unli captures a more probabilistic notion of inference, it targets average responses and does not attempt to model the full range of interpretations. zhang and de marneffe (2021) contrast systematic inference (high agreement) and ambiguous cases (high disagreement). they build artificial annotators using bert @xcite to simulate annotation variation, enabling downstream models to determine if a given premise-hypothesis pair is likely to elicit disagreement. @xcite further improve modeling of opinion distributions beyond standard softmax assumptions. together, these studies mark a shift from a prescriptive view-assuming a single correct label-toward a descriptive approach that acknowledges interpretive variations. they help pave the way for systems that capture ambiguity inherent in natural language. however, simply modeling disagreement alone does not explain why interpretations diverge. to advance beyond descriptive modeling, we argue that nli systems must also systematically identify and categorize the sources of disagreementespecially content-based ambiguity-as a foundation for more perspective-sensitive inference @xcite . we next examine the sources that give rise to divergent judgments in nli. ## disagreement sources in nli according to the "triangle of reference" @xcite , disagreement in annotation arises from three main sources: (1) interpretative ambiguity in the input content itself (uncertainty in sentence meaning); (2) unclear annotation guidelines (underspecification in guidelines); and (3) differences in annotators' background knowledge or task understanding (annotator behavior). this framework maps directly onto annotation workflows in nli benchmarks. building on this foundation, @xcite propose a more fine-grained taxonomy for nli, refining each category into subtypes that reflect recurring premisehypothesis patterns (figure 2 ). nli annotation disagreement sources lexical coreference underspecification of guidelines implicature presupposition probabilistic enrichment imperfection temporal reference interrogative hypothesis high overlap ambiguity in sentence meaning annotator behavior accommodating minimally added content figure 2: the taxonomy of disagreement sources developed by jiang and de marneffe (2022), building on aroyo and welty (2015). while these frameworks classify sources of disagreement, we argue for reframing such disagreement as a signal of coexisting interpretations to model-not noise to be resolved. in the subsections below, we adopt a variant of this taxonomy, reframing uncertainty in sentence meaning as ambiguity in sentence meaning (section 3.1), a shift already noted in figure 2 . this distinction is central to our position. for completeness, we also briefly describe the roles of guideline underspecification (section 3.2) and annotator behavior (section 3.3), though these are not the central emphasis of our position. in addition, while we view jiang and de marneffe's taxonomy as a valuable classification framework, we go further: rather than treating disagreement as noise to be explained or resolved, we reframe it as a meaningful signal of coexisting interpretations-something to be modeled directly as part of the nli task. ## why does content-based ambiguity deserve special attention? among the disagreement sources outlined above, content-based ambiguity stands out as the only type that can be systematically addressed through computational modeling without relying on additional information about the annotators or the guidelines they follow. this form of ambiguity originates from the text itself, independent of the annotation process, yet it remains a fundamental driver of divergent interpretations. as such, it represents a root cause of disagreement inherent to natural language, posing a persistent challenge for inference systems aiming for consistent and reliable predictions. implicature ambiguity premise: "it hopes to bring on another 25 or 35 people when the new building opens next fall." hypothesis: "they already have a waiting list for the new building" figure 6 : implicature ambiguity: annotators may infer a waiting list from hopes to bring on another 25 or 35 people. if so, they label it entailment; if not, they label neutral. consider the implicature ambiguity in figure 6 . some annotators interpret hopes to bring on another 25 or 35 people as implying a waiting list and choose entailment. others focus strictly on what is stated and select neutral. this interpretative variability stems from linguistic ambiguity rather than annotator background or faulty instructions. such cases underscore the importance of treating content-based ambiguity as central to analysis, rather than dismissed as noise. jiang and de marneffe (2022)'s findings indicate that the most common sources of disagreement fall under content-based ambiguity, underscoring its prevalence. in contrast, issues related to underspecified guidelines can typically be resolved through clearer instructions and better annotation practices, meaning they do not strongly reflect genuine differences in human interpretation. similarly, while some annotator behaviorssuch as accommodating minimally added content-reflect natural variation, others like high overlap may undermine the goals of the nli task. disagreements stemming from annotator behavior or guideline underspecification therefore warrant scrutiny before being treated as meaningful. not all disagreements are noise, though some clearly reflect error @xcite . in contrast, content-based ambiguity arises from the language itself and requires no external filtering or supervision, making it a uniquely reliable source of interpretive variation. identifying such ambiguity supports the creation of disambiguated versions, allowing nli benchmarks to better capture the range of plausible interpretations. beyond benchmarking, ambiguity-aware modeling has practical consequences in downstream settings. identifying content-based ambiguity in nli data has significant real-world implications. nli frequently serves as a core component of factverification pipelines, where it is used to assess the relationship between claims and supporting evidence @xcite . effectively pinpointing potential ambiguities-including those intentionally introduced-strengthens such pipelines by improving their capacity to flag potentially misleading content in real-world settings @xcite . however, there are currently no established methods for disentangling disagreements caused by content-based ambiguity from those arising due to underspecified annotation guidelines or annotator behavior. as a result, most existing work focuses primarily on detecting premise-hypothesis pairs with high annotation disagreement, rather than investigating the underlying types of disagreement, particularly those stemming from ambiguity. therefore, there is a necessity to build models that: (1) identify ambiguous premise-hypothesis pairs and (2) classify the respective ambiguity type. these observations motivate two concrete tasksambiguity detection and ambiguity classificationwhich we discuss next. while some efforts have been made to develop models that detect instances with high annotator disagreement @xcite @xcite , there are no existing implementations that specifically identify or classify ambiguous instances in nli-underscoring the need for systems designed to address this gap. ## call to action: recognizing ambiguity as signal, not noise disagreement among annotators in natural language inference (nli) is often treated as noisesomething to minimize or discard. however, many of these disagreements reflect genuine interpretive differences, often triggered by ambiguity in the premise, hypothesis, or both. our analysis suggests that while underspecified annotation guidelines and inconsistent annotator behavior can lead to label disagreement, such cases must be carefully scrutinized to distinguish between annotation errors and genuine differences in human interpretation. in contrast, ambiguity in the nli input itselfwhether lexical, syntactic, semantic, or pragmaticserves as a clear, process-independent signal of interpretive variation, providing a basis for understanding how meaning can diverge across readers. this highlights a needed shift: from optimizing for annotator consensus to explicitly identifying and characterizing ambiguity as a central feature of natural language. to support this shift, we outline the following two key directions: a key novelty of this work lies in articulating a unified framework (figure 1 ) that extends beyond prior approaches focused solely on modeling annotation distribution. whereas earlier efforts largely stop at detecting high-disagreement instances, this framework explicitly distinguishes ambiguity from other sources of variation by leveraging linguistic features and pertinent background knowledge. the framework consists of four stages. the first determines whether a premise-hypothesis pair is inherently ambiguous, thereby distinguishing between genuine interpretive variation from annotation noise and other sources of disagreement. if an instance is ambiguous, the second classifies the type(s),creating systematic linkages to linguistic background knowledge. the third stage generates relevant disambiguated versions, after which the fourth models inference classification for both ambiguous and nonambiguous instances, based on predictions from earlier stages. the framework's strength lies in offering a structured and operational foundation for ambiguity-aware nli, moving the field from descriptive accounts of disagreement toward a principled methodology that can be empirically tested. by pursuing these goals, we can build nli models that are not only more aligned with human interpretation, but also more explainable in predictions. ambiguity-aware systems better align with human interpretation and produce more consistent, interpretable, and robust predictions. this reframing is not only timely-it is essential for developing nli systems that reflect the complexity of human understanding, rather than abstracting it away. while we advocate for a shift toward ambiguityaware nli systems, realizing this vision is currently constrained by a key limitation: the lack of datasets that are explicitly annotated for ambiguity and categorized by ambiguity type. most existing nli datasets are not designed with interpretive variation or ambiguity classification in mind, making it difficult to systematically identify and analyze ambiguous instances or to evaluate models on their ability to handle them. this gap limits the development and benchmarking of methods for detecting and classifying ambiguity. the absence of gold-standard annotations for different ambiguity types hinders progress in training and evaluating models that aim to align more closely with human interpretive processes. to address this, we suggest two complementary directions. first, there is a clear need for the creation of new datasets specifically annotated for ambiguity presence and type. such resources would lay the groundwork for both empirical analysis and model development. second, we see promise in exploring unsupervised or weakly supervised methods that can surface potential ambiguities without requiring extensive manual labeling. techniques leveraging patterns of annotator disagreement, discourse features, or model uncertainty could offer scalable alternatives in the absence of annotated data. despite current limitations, these strategies offer a promising path toward building nli systems that better reflect the complexity and nuance of human language understanding.