Are Natural Language Inference Models IMPPRESsive ? L earning IMPlicature and PRESupposition (2020)

Abstract: Natural language inference (NLI) is an increasingly important task for natural language understanding, which requires one to infer whether a sentence entails another. However, the ability of NLI models to make pragmatic inferences remains understudied. We create an IMPlicature and PRESupposition diagnostic dataset (IMPPRES), consisting of 32K semi-automatically generated sentence pairs illustrating well-studied pragmatic inference types. We use IMPPRES to evaluate whether BERT, InferSent, and BOW NLI models trained on MultiNLI (Williams et al., 2018) learn to make pragmatic inferences. Although MultiNLI appears to contain very few pairs illustrating these inference types, we find that BERT learns to draw pragmatic inferences. It reliably treats scalar implicatures triggered by “some” as entailments. For some presupposition triggers like “only”, BERT reliably recognizes the presupposition as an entailment, even when the trigger is embedded under an entailment canceling operator like negation. BOW and InferSent show weaker evidence of pragmatic reasoning. We conclude that NLI training encourages models to learn some, but not all, pragmatic inferences.

## introduction one of the most foundational semantic discoveries is that systematic rules govern the inferential relationships between pairs of natural language sentences (aristotle, de interpretatione, ch. 6). in natural language processing, natural language inference (nli)-a task whereby a system determines whether a pair of sentences instantiates in an entailment, a contradiction, or a neutral relation-has been useful for training and evaluating models on sentential reasoning. however, linguists and philosophers now recognize that there * equal contribution are separate semantic and pragmatic modes of reasoning @xcite @xcite @xcite , and it is not clear which of these modes, if either, nli models learn. we investigate two pragmatic inference types that are known to differ from classical entailment: scalar implicatures and presuppositions. as shown in figure 1 , implicatures differ from entailments in that they can be denied, and presuppositions differ from entailments in that they are not canceled when placed in entailment-cancelling environments (e.g., negation, questions). to enable research into the relationship between nli and pragmatic reasoning, we introduce imppres, a fine-grained nli-style diagnostic test dataset for probing how well nli models perform implicature and presupposition. containing 25.5k sentence pairs illustrating key properties of these pragmatic inference types, imppres is automatically generated according to linguist-crafted templates, allowing us to create a large, lexically varied, and well controlled dataset targeting specific instances of both types. we first investigate whether presuppositions and implicatures are present in nli models' training data. we take multinli @xcite as a case study, and find it has few instances of pragmatic inference, and almost none that arise from specific lexical triggers (see §4). given this, we ask whether training on multinli is sufficient for models to generalize about these largely absent commonsense reasoning types. we find that generalization is possible: the bert nli model shows evidence of pragmatic reasoning when tested on the implicature from some to not all, and the presuppositions of certain triggers (only, cleft existence, possessive existence, questions). we obtain some negative results, that suggest that models like bert still lack a sophisticated enough understanding of the meanings of the lexical triggers for implicature and presupposition (e.g., bert treats several word pairs as synonyms, e.g., most notably, or and and). our contributions are: (i) we provide a new diagnostic test set to probe for pragmatic inferences, complete with linguistic controls, (ii) to our knowledge, we present the first work evaluating deep nli models on specific pragmatic inferences, and (iii) we show that bert models can perform some types of pragmatic reasoning very well, even when trained on nli data containing very few explicit examples of pragmatic reasoning. we publicly release all imppres data, models evaluated, annotations of multinli, and the scripts used to process data. 1 ## background: pragmatic inference we take pragmatic inference to be a relation between two sentences relying on the utterance context and the conversational goals of interlocutors. pragmatic inference contrasts with semantic entailment, which instead captures the logical relationship between isolated sentence meanings @xcite . we present implicature and presupposition inferences below. ## annotating multinli for pragmatics in this section, we present results of an annotation effort that show that multinli contains very little explicit evidence of pragmatic inferences of the type tested by imppres. although @xcite report that 22% of the multinli development set sentence pairs contain lexical triggers (such as regret or stopped) in the premise and/or hypothesis, the mere presence of presuppositiontriggering lexical items in the data does not show that multinli contains evidence that presuppositions are entailments, since the sentential inference may focus on other types of information. to address this, we randomly selected 200 sentence pairs from the multinli matched development set and presented them to three expert annotators with a combined total of 17 years of training in formal semantics and pragmatics. 4 annotators answered the following questions for each pair: (1) are the sentences p and h related by a presupposition/implicature relation (entails/is en4 the full annotations are on the imppres repository. agreement among annotators was low, suggesting that few multinli pairs are paradigmatic cases of implicatures or presuppositions. we found only 8 presupposition pairs and 3 implicature pairs on which two or more annotators agreed. moreover, we found only one example illustrating a particular inference type tested in imppres (the presupposition of possessed definites). all others were tagged as existence presuppositions and conversational implicatures (i.e. loose inferences dependent on world knowledge). the union of annotations was much larger: 42% of examples were identified by at least one annotator as a presupposition or implicature (51 presuppositions and 42 implicatures, with 10 sentences receiving divergent tags). however, of these, only 23 presuppositions and 19 implicatures could reliably be used to learn pragmatic inference (in 14 cases, the given tag did not match the pragmatic inference, and in 27 cases, computing the inference did not affect the relation type). again, the large majority of implicatures were conversational, and most presuppositions were existential, and generally not linked to particular lexical triggers (e.g., topic marking). we conclude that the multinli dataset at best contains some evidence of loose pragmatic reasoning based on world knowledge and discourse structure, but almost no explicit information relevant to lexically triggered pragmatic inference, which is of the type tested in this paper. generating data lets us control the lexical and syntactic content so that we can guarantee that the sentence pairs in imppres evaluate the desired phenomenon (see @xcite , for related discussion). furthermore, the codebase we use allows for greater lexical and syntactic variety than in many other templatic datasets (see discussion in @xcite . one limitation of this methodology is that generated sentences, while generally grammatical, often describe highly unlikely scenarios, or include low frequency combinations of lexical items (e.g., sabrina only reveals this pasta). another limitation is that generated data is of limited use for training models, since it contains simple regularities that supervised classifiers may learn to exploit. thus, we create imp-pres solely for the purpose of evaluating nli models trained on standard datasets like multinli. ## general discussion & conclusion we observe some encouraging results in §6-7. we find strong evidence that bert learns scalar implicatures associated with determiners some and all. pragmatic or logical reasoning was not diagnosable for the other scales, whose meaning was not fully understood by our models (as most scalar pairs were treated as synonymous). in the case of presuppositions, the bert nli models, and bow to some extent, perform well on a number of our subdatasets (only, cleft existence, possessive existence, questions). for the other subdatasets, the models did not perform as expected on the basic unembedded presupposition triggers, again sug-gesting the model's lack of knowledge of the basic meaning of these words. though their behavior is far from systematic, this is suggestive evidence that some nli models can perform in ways that correlate with human-like pragmatic behavior. given that multinli contains few examples of the type found in imppres (see §4), where might our positive results come from? there are two potential sources of signal for the bert model: nli training, and pretraining (either bert's masked language modeling objective or its input word embeddings). nli training provides specific examples of valid (or invalid) inferences constituting an incomplete characterization of what commonsense inference is in general. since presuppositions and scalar implicatures triggered by specific lexical items are largely absent from the multinli data used for nli training, any positive results on imppres would likely use prior knowledge from the pretraining stage to make an inductive leap that pragmatic inferences are valid commonsense inferences. the natural language text used for pretraining certainly contains pragmatic information, since, like any natural language data, it is produced with the assumption that readers are capable of pragmatic reasoning. maybe this induces patterns in the data that make the nature of those assumptions recoverable from the data itself. this work is an initial step towards rigorously investigating the extent to which nli models learn semantic versus pragmatic inference types. we have introduced a new dataset imppres for probing this question, which can be reused to evaluate pragmatic performance of any nli given model.