Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences (2022)

Abstract: Presuppositions are assumptions that are taken for granted by an utterance, and identifying them is key to a pragmatic interpretation of language. In this paper, we investigate the capabilities of transformer models to perform NLI on cases involving presupposition. First, we present simple heuristics to create alternative “contrastive” test cases based on the ImpPres dataset and investigate the model performance on those test cases. Second, to better understand how the model is making its predictions, we analyze samples from sub-datasets of ImpPres and examine model performance on them. Overall, our findings suggest that NLI-trained transformer models seem to be exploiting specific structural and lexical cues as opposed to performing some kind of pragmatic reasoning.

## introduction natural language inference (nli) is the task of predicting whether a sentence entails, contradicts or is neutral with respect to another sentence. while nli is presented as a task with the goal of pushing the frontier of language understanding, one question that remains elusive is the extent to which learning models trained on nli datasets acquire competence in pragmatic reasoning. for example, "rose's drawing stunned the audience" presupposes that "rose has a drawing." in terms of the nli task, the relationship between these two statements is one of entailment. however, "rose stunned the audience" does not trigger the same presupposition. if an nli system relies on an incorrect heuristic to predict entailment, such as the fact that both statements have "rose" in them, the system would fail on the modified version. while there exists a plethora of nli datasets (e.g. snli @xcite , mnli @xcite , hans @xcite ), issues related to pragmatics remain under-explored. one exception is the imppres dataset @xcite , which contains english nli instances involving two pragmatic phenomena: implicature and presupposition. in linguistics and philosophy of language, presuppositions are assumptions and beliefs that are shared and taken for granted by discourse participants without an explicit mention in the discourse context. presuppositions are prevalent in language and understanding them facilitates smooth communication and is crucial for a proper understanding of the meaning being conveyed in a given context. in this work, we investigate how well transformer models perform on cases from imppres involving presupposition. we ask the following question: in cases where these models seem to be doing well in terms of (nli) accuracy performance, are they exploiting any patterns or correlations between certain words and decision labels which happen to lead to a good performance on imppres, but which might not generalize beyond it? our contribution is two-fold: first, we present simple heuristics to create alternative "contrastive" test cases based on the presupposition sub-datasets in imppres and investigate the model performance on those test cases. second, from those datasets, we draw a random set of testing samples and examine the performance of the models on these sets. in each of those cases, we identify specific patterns or cues that seem to be influencing the accuracy performance (in either the correct or incorrect predictions). our findings suggest that transformer-based nli models seem to be exploiting surface-level lexical and structural cues as opposed to performing some kind of pragmatic reasoning and that these cues are highly task-specific and do not generalize to adversarially perturbed versions of the input. ## related work natural language inference (nli) is the task of predicting whether one passage entails another. in our work, we adopt the modern formulation 1 cast as a three-way classification task where two statements, a premise and a hypothesis, are bound by a relationship of entailment, contradiction or neither (the "neutral" case). many datasets have been introduced for the task including snli @xcite , mnli @xcite , mpe @xcite , xnli @xcite , etc. the imppres dataset @xcite ) has been proposed to focus specifically on presupposition and implicature. imppres consists of 25.5k sentence pairs further divided into several sub-datasets, each focusing on a specific type of implicature or presupposition. in our work, we focus on the presupposition part of imppres. other work studied the same general issue of checking whether systems systematically changed their labels on perturbed samples in an expected way @xcite @xcite @xcite . the hans dataset @xcite in particular targets issues where nli models fail because they rely on superficial syntactic heuristics. unlike our work, the nli instances they examineand nli in general-often require additional world knowledge, or linguistic knowledge that is highly lexicalized (e.g., a list of factive verbs). by contrast, the class of presuppositions that we examine is tied to specific linguistic constructions that systematically trigger them, such as definite articles and cleft constructions, which could potentially be easier. we nevertheless show that the models we tested fail to generalize systematically on presuppositional inferences, complementing previous results. also relevant to this work is the commitment-bank dataset @xcite which fo-1 for earlier (two-way classification) formulations, the reader is referred to @xcite and @xcite cuses on issues involving presupposition projection in various environments. this work highlights that projection is hardly a binary affair, and that judgments about entailment (roughly corresponding to speakers' commitment in the language of commit-mentbank) are graded. ## methodology our study consists of fine-tuning a pre-trained transformer model (either bert @xcite or roberta @xcite ), then evaluating its performance on the various presupposition subdatasets of imppres in addition to perturbed versions of those sub-datasets introduced in this work. we fine-tune on mnli, following @xcite . we conduct two studies in order to investigate the performance of these models and better understand their capabilities in making pragmatic inferences. in the first study, we present simple heuristics to create "contrastive" test cases. we then evaluate the finetuned model on those test cases quantitatively. in the second study, we draw from each sub-dataset of imppres a random set of 100 samples. we examine these sets for structural and lexical cues that might be influencing the performance of bert. ## study 1: contrastive probing in this study, we present simple heuristics to create "contrastive" test cases for the various imppres sub-datasets and investigate model performance on those test cases. we divide our probes into two types: those involving contrastive sentences which are grammatical english sentences, and those with corrupt input which are ungrammatical. ## study 2: sample error analysis we examine bert's performance on 100 samples randomly drawn from each sub-dataset. we see patterns repeating across several datasets-all about surface cues which should not be directly responsible for presupposition. we group our insights into themes. ## conclusion we investigated bert's capabilities to perform nli on cases involving presupposition. our analysis suggests that nli-trained bert exploits lexical and structural cues to do so, and that these cues are highly task-specific and do not generalize to adversarially perturbed versions of the input.