Identifying inherent disagreement in natural language inference (2021)

Abstract: Natural language inference (NLI) is the task of determining whether a piece of text is entailed, contradicted by or unrelated to another piece of text. In this paper, we investigate how to tease systematic inferences (i.e., items for which people agree on the NLI label) apart from disagreement items (i.e., items which lead to different annotations), which most prior work has overlooked. To distinguish systematic inferences from disagreement items, we propose Artificial Annotators (AAs) to simulate the uncertainty in the annotation process by capturing the modes in annotations. Results on the CommitmentBank, a corpus of naturally occurring discourses in English, confirm that our approach performs statistically significantly better than all baselines. We further show that AAs learn linguistic patterns and context-dependent reasoning.

## introduction learning to effectively understand unstructured text is integral to natural language understanding (nlu), covering a wide range of tasks such as question answering, semantic textual similarity and sentiment analysis. natural language inference (nli), an increasingly important benchmark task for nlu research, is the task of determining whether a piece of text is entailed, contradicted by or unrelated to another piece of text (i.a., @xcite . @xcite observed inherent disagreements among annotators in several nli datasets, which cannot be smoothed out by hiring more people. they pointed out that to achieve robust nlu, we need to be able to tease apart systematic inferences (i.e., items for which most people agree on the annotations) from items inherently leading to disagreement. the last example in while others view it as a contradiction (-3). a common practice to generate an inference label from annotations is to take the average (i.a., @xcite . in this case, the average of the annotations is 0.25 and the gold label for this item would thus be "neutral", but such label is not accurately capturing the annotation distribution. alternatively, some work simply ignores items on which annotators disagree and only studies systematic inference items @xcite . here, we aim at teasing apart systematic inferences from inherent disagreements. for nli: teasing disagreement items, labeled "disagreement", from systematic inferences, which can be "contradiction", "neutral" or "entailment". to this end, we propose artificial annotators (aas), an ensemble of bert models @xcite , which simulate the uncertainty in the annotation process by capturing modes in annotations. that is, we expect to utilize simulated modes of annotations to enhance finer-grained nli label prediction. our results, on the commitmentbank, show that aas perform statistically significantly better than all baselines (including bert baselines) by a large margin in terms of both f1 and accuracy. we also show that aas manage to learn linguistic patterns and context-dependent reasoning. ## data: the commitmentbank the commitmentbank (cb) is a corpus of 1,200 naturally occurring discourses originally collected from news articles, fiction and dialogues. each discourse consists of up to 2 prior context sentences and 1 target sentence with a clause-embedding predicate under 4 embedding environments (negation, modal, question or antecedent of conditional). annotators judged the extent to which the speaker/author of the sentences is committed to the truth of the content of the embedded clause (cc), responding on a likert scale from +3 to -3, labeled at 3 points (+3/speaker is certain the cc is true, 0/speaker is not certain whether the cc is true or false, -3/speaker is certain the cc is false). following jiang and de marneffe (2019b), we recast cb by taking the context and target as the premise and the embedded clause in the target as the hypothesis. common nli benchmark datasets are snli @xcite and multinli @xcite , but these datasets have only one annotation per item in the training set. cb has at least 8 annotations per item, which permits to identify items on which annotators disagree. jiang and de marneffe (2019b) discarded items if less than 80% of the annotations are within one of the following three ranges: [1,3] entailment, 0 neutral, [-3,-1] contradiction. the gold label for example 3 in ## model: artificial annotators we aim at finding an effective way to tease items leading to systematic inferences apart from items leading to disagreement. as pointed out by @xcite , annotated labels are subject to uncertainty. annotations are indeed influenced by several factors: workers' past experience and concentration level, cognition complexities of items, etc. they proposed to simulate the annotation process in an active learning paradigm to make use of the annotations that contribute to uncertainty. likewise, for nli, @xcite observed that directly training on raw annotations using annotator 2020 ) used a mixed-effect model to learn a mapping from an item and the associated annotator identifier to a nli label. however, annotator identifiers are not always accessible, especially in many datasets that have been there for a while. thus, we decide to simulate the annotation process instead of learning from real identifiers. as shown by @xcite , if annotations of an item follow unimodal distributions, then it is suitable to use aggregation (i.e., take an average) to obtain a inference label; but such an aggregation is not appropriate when annotations follow multi-modal distributions. without loss of generality, we assume that items are associated with n-modal distributions, where @xmath1. usually, systematic inference items are tied to unimodal annotations while disagreement items are tied to multi-modal annotations. we, thus, introduce the notion of artificial annotators (aas), where each individual "annotator" learns to model one mode. ## experiment we include five baselines to compare with: ## analysis aas outperform all bert-based models under all classes. however, compared with the heuristic, aas show an inferior result on "neutral" items mainly due to the lack of "neutral" training data. the first 4 examples in ## conclusion we introduced finer-grained natural language inference. this task aims at teasing systematic inferences from inherent disagreements, overlooked in prior work. we show that our proposed aas, which simulate the uncertainty in annotation process by capturing the modes in annotations, perform statistically significantly better than all baselines. however the best performance obtained (∼66%) is still far from achieving robust nlu, leaving room for improvement.