Uncertain Natural Language Inference (2020)

Abstract: We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments. We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically-labeled NLI data can be used in pre-training. Our best models correlate well with humans, demonstrating models are capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks.

## introduction variants of entailment tasks have been used for decades in benchmarking systems for natural language understanding. recognizing textual entailment (rte) or natural language inference (nli) is traditionally a categorical classification problem: predict which of a set of discrete labels apply to an inference pair, consisting of a premise ( ) and hypothesis (ℎ). the fracas consortium offered the task as an evaluation mechanism, along with a small challenge set @xcite , which was followed by the rte challenges @xcite . despite differences between these and recent nli datasets @xcite @xcite , nli hsa remained a categorical prediction problem. however, entailment inference is uncertain and has a probabilistic nature @xcite . maintaining nli as a categorical classification problem is not ideal since coarse categorical labels mask the uncertain and probabilistic nature of entailment inference. nli pairs may share a coarse label, but the probabilities that the hypotheses are entailed by their corresponding premises may vary greatly (see we propose uncertain natural language inference (unli), a refinement of nli that captures more subtle distinctions in meaning by shifting away from categorical labels to the direct prediction of human subjective probability assessments. we illustrate that human-elicited probability assessments contain subtle distinctions on the likelihood of a hypothesis conditioned on a premise, and unli captures these distinctions far beyond categorical labels in popular nli datasets. we demonstrate how to elicit unli annotations. using recent large-scale language model pre-training, we provide experimental results illustrating that systems can often predict unli judgments, but with clear gaps in understanding. we conclude that scalar annotation protocols should be adopted in future nli-style dataset creation, which should enable new work in modeling a richer space of interesting inferences. ## eliciting unli annotations we elicit subjective probabilities from crowdsource workers (mturk) for premise-hypothesis pairs from existing nli data. annotators are asked to estimate how likely the situation described in the hypothesis sentence would be true given the premise. following the efficient annotation of scalar labels framework (easl; sakaguchi and durme, 2018), we present annotators 5 sentence-pairs, each with a slider bar enabling direct assessment for each pair and ask annotators to calibrate their score for a sentence-pair based on the scores they provided to the other four pairs. 1 in contrast to the uniform scale employed in the original easl protocol, we modify the interface to allow finer-grained values near 0.0 and 1.0, following psychological findings that humans are especially sensitive to values near the ends of the probability spectrum @xcite ). 2 this interface decision is a key distinction of this work contrasting prior efforts that averaged likertscale (ordinal) annotations. this allows us to capture the difference between nli pairs that are both appropriately contradicted or entailed under nli, but that have a perceived difference of less than 1% probability. in order to capture the sensitivity near these ends, we adopt a more fine-grained slider bar with 10,000 steps with a logistic transformation. specifically, for raw @xmath0, 10000], we apply a scaled logistic function ( ) = ( ( -5000)) to re-scale the final result range to [0, 1]. we ran pilots to tune , and determine that people tend to choose much lower probability for some events even though they are just slightly less likely (e.g., just below 50%). 3 1 example pairs were provided in the instructions along with suggested probability values. see appendix a for details of the annotation interface and qualifications. data we annotate, i.e. elicit a @xmath1, 1], for a subset of snli @xcite examples and refer to this data as u-snli. foot_0 snli's training set contains 7,931 distinct premises paired with at least 5 distinct neutral (neu) hypotheses. for each premise, we sample 5 neutral hypotheses, resulting in 39,655 of these neu pairs annotated. an additional 15,862 contradicted (con) and entailed (ent) pairs are annotated for our training set, resulting in 55,517 training examples. for our dev and test sets, we respectively annotated 3,040 examples sampled from snli's dev and test splits. in total, we annotated 61,597 examples, about 12% of all examples in snli. figure 1 plots the resultant median and quartile for each categorical snli label in the u-snli dev set, showing the wide range of probability judgments elicited for each label (see ## prediction formally, given a @xmath2, a unli model : p × @xmath3, 1] should output an uncertainty score ˆ ∈ [0, 1] of the premise-hypothesis pair that correlates well with a human-provided subjective probability assessment. we train a regression unli model to predict the probability that a premise entails a hypothesis. we modify the sentence pair classifier 6 in bert to exploit recent advancements in large-scale language model pre-training. following devlin et al. (2019) , we concatenate the premise and the hypothesis, with a special sentinel token (cls) inserted at the beginning and a separator (sep) inserted after each sentence, tokenized using wordpiece. after encoding the concatenated token sequence with bert, we take the encoding of the first sentinel token. we pass the resulting feature vector f ( , ℎ) through a sigmoid-activated linear layer to obtain a probability, instead of a softmax used in categorical nli. we directly model unli as a regression problem, trained using a binary cross-entropy loss 7 between the human annotation and the model output ˆ . owing to the concerns raised with annotation artifacts in snli @xcite @xcite , we include a hypothesis-only baseline. 8 metrics we compute pearson correlation ( ), the spearman rank correlation ( ), and the mean square error (mse) between y and ˆ as the metrics to measure the to performance of unli models. pearson measures the linear correlation between the gold probability assessments and model's output; spearman measures the ability of the model ranking the premise-hypothesis pairs with 6 the neural architecture for multinli @xcite in @xcite . 7 no significant difference is observed with an 2 loss. 8 see appendix d for additional training details. ## results & analysis qualitative error analysis pre-training with snli can we leverage the remaining roughly 500,000 snli training pairs that only have categorical labels? one method would be to train a categorical nli model on snli and when fine-tuning on u-snli, replace the last layer of the network from a categorical prediction with a sigmoid function. 11 however, a typical categorical loss function would not take into account the ordering between the different categorical labels. 12 instead, we derive a surrogate function : @xmath4, 1] that maps snli categorical @xmath5, neu, con} to the average score of all u-snli training annotations labeled with in snli. we use this mapping to pre-train a regression model on the snli training examples not included in u-snli. we also fine-tune the model on u-snli's training set. ## conclusion we proposed uncertain natural language inference (unli), a new task of directly predicting human likelihood judgments on nli premisehypothesis pairs. in short, we have shown that not all nli contradictions are created equal, nor neutrals, nor entailments. we demonstrated that (1) eliciting supporting data is feasible, and (2) annota-tions in the data can be used for improving a scalar regression model beyond the information contained in existing categorical labels, using recent contextualized word embeddings, e.g. bert. humans are able to make finer distinctions between meanings than is being captured by current annotation approaches; we advocate the community strives for systems that can do the same, and therefore shift away from categorical nli labels and move to something more fine-grained such as our unli protocol.