PROPRES : Investigating the Projectivity of Presupposition with Various Triggers and Environments (2023)

Abstract: What makes a presupposition of an utterance —information taken for granted by its speaker— different from other pragmatic inferences such as an entailment is projectivity (e.g., the negative sentence the boy did not stop shedding tears presupposes the boy had shed tears before). The projectivity may vary depending on the combination of presupposition triggers and environments. However, prior natural language understanding studies fail to take it into account as they either use no human baseline or include only negation as an entailment-canceling environment to evaluate models’ performance. The current study attempts to reconcile these issues. We introduce a new dataset, projectivity of presupposition (PROPRES), which includes 12k premise–hypothesis pairs crossing six triggers involving some lexical variety with five environments. Our human evaluation reveals that humans exhibit variable projectivity in some cases. However, the model evaluation shows that the best-performed model, DeBERTa, does not fully capture it. Our findings suggest that probing studies on pragmatic inferences should take extra care of the human judgment variability and the combination of linguistic items.

## introduction it is an open question as to whether language models can learn a human-like pragmatic inference @xcite . a speaker does not always explicitly say everything in an utterance, but a hearer can infer what is implicit in it. one notable case concerns a presupposition that refers to information taken for granted by a speaker of an utterance @xcite . presuppositions are prevalent in our everyday communication; hence, a comprehensive investigation of whether models can understand them in the same way as humans can contribute to the development of a better language understanding system. presupposition triggers introduce presuppositions (e.g., again in figure 1 (a) ). a presuppo-(f) the doctor had cut the tree before. sition of (a) is the doctor had cut the tree before (f). what makes the presupposition different from an entailment (in this case, the doctor cut the tree one more time) is projectivity: the presupposition projects out of entailment-canceling environments (e.g., negative (b), interrogative (c), conditional (d), and modal (e) sentences) while the entailment does not. 1 in other words, the presupposition (f) holds in the environments (b-e), but the entailment (the doctor cut the tree one more time) does not. crucially, linguistic studies suggest that the projectivity can vary depending on many factors @xcite @xcite @xcite . previous probing studies in natural language processing examine models' performance on presuppositions in the natural language inference (nli) task @xcite . however, they do not fully take into account the variable aspect of the projectivity. for instance, @xcite obtain no human baseline, which makes models' performance hard to ## experiment 1: reevaluating imppres one limitation in @xcite is no human evaluation, which leaves it open whether models capture any variable projectivity exhibited by humans. to overcome it, we collect human labels on a subset of imppres, testing the performance of the two models, roberta and deberta, against the human results. ## experiment 2: propres an investigation of the projectivity of presupposition with imppres is far from comprehensive because we can find more triggers in the literature (e.g., 27 triggers in @xcite and @xcite in total) and none of the six triggers which we analyze in imppres has lexical variation. using six additional triggers with some lexical variety, we create a new dataset, propres, which allows us to investigate the variable projectivity and models' behavior more comprehensively. ## conclusion our experiments reveal that humans exhibit the variable projectivity of presupposition in some conditions (two out of 24 and six out of 24 conditions in experiments 1 and 2, respectively), but the bestperformed model, deberta, does not capture it most of the time, indicating that it does not generalize pragmatic inferences for presuppositions. in our experiments, quite a few conditions are excluded from the analysis for various reasons such as lexical ambiguity in some items, disagreements in human judgments, and the models' lack of knowledge. to tease apart these factors carries us well beyond the scope of this study. however, this fact suggests that we need to be careful with dataset creation so that we can train or evaluate models on well-designed datasets targeting pragmatic inferences.