Pragmatic and Logical Inferences in NLI Systems: The Case of Conjunction Buttressing (2022)

Abstract: An intelligent system is expected to perform reasonable inferences, accounting for both the literal meaning of a word and the meanings a word can acquire in different contexts. A specific kind of inference concerns the connective and, which in some cases gives rise to a temporal succession or causal interpretation in contrast with the logic, commutative one (Levinson, 2000). In this work, we investigate the phenomenon by creating a new dataset for evaluating the interpretation of and by NLI systems, which we use to test three Transformer-based models. Our results show that all systems generalize patterns that are consistent with both the logical and the pragmatic interpretation, perform inferences that are inconsistent with each other, and show clear divergences with both theoretical accounts and humans’ behavior.

## introduction implicature is the term used in semantics and pragmatics to describe an inference that goes beyond the literal sense of what is said. implicatures have received relatively limited attention in computational linguistics, since they are highly dependent on the communication context and on commonsense knowledge. however, the notion of generalized conversational implicature (gci) @xcite captures the fact that some of these meaning enrichments are more general than others: they are still dependent on context, but they are also strongly conventionalized and they act as default inferences, which are carried out unless canceled by additional contextual information. with this study, we aim at contributing to the research on gcis in nlp systems by focusing on a specific type of gci, namely levinson's iimplicatures associated with the conjunction and @xcite . studies have noted that and is regularly interpreted as a temporal succession or causal connective (from john repaired the engine and the car started we understand that the car started as a result of john repairing the engine) @xcite . this implicature, which is referred to as conjunction buttressing by @xcite , contradicts the commutative interpretation of and traditionally assumed in formal logic and semantics: if a and b entails b after a, a and b is not equivalent to b and a. moreover, the implicature takes place only when the conjuncts express dynamic events, while with static ones and preserves the commutative property (e.g., john was awake and the dog slept entails the dog slept and john was awake). to address the problem of the scarcity of data for the study of gcis and conjunction buttressing in particular, we created a dataset for the study of the interpretation of and by nli systems, using manual annotation to obtain quality data and control for features relevant for the implicature according to theoretical accounts. we assigned two different label sets based on a pragmatic hypothesis (and triggers the implicature) and a logic one (and is commutative), to distinguish logical vs. pragmatic behavior of the systems. we tested three transformer-based nli systems fine-tuned on mnli @xcite on our dataset. we identified systematic inference patterns involving the interpretation of and that are common to all three systems. some of these patterns are in accordance with the pragmatic hypothesis and others with the logic one. we found that the systems make inferences that are inconsistent with each other, and in many cases their interpretation of and is different from both the human interpretation and theoretical accounts. to see whether the results are due to biases in the systems' training set, we ran an analysis of mnli aimed at identifying inference patterns involving and that are used by annotators, finding that the inferences generalized by systems are exemplified to varying degrees. after describing related work in section 2, in section 3 we describe how we collected data to assess logical and pragmatic interpretations in nli systems. 1 results of the experiments with nli systems are illustrated in section 4, along with the analysis on mnli and the results of a human behavioral study. conclusions are devoted to suggestions for future work and to the discussion of the limitations of the present work. by highlighting limitations of current systems on our dataset, we argue for a stronger convergence of neural systems for inference and cognitive models of gcis. ## related work previous nlp studies on implicatures mostly focused on scalar implicatures, inferences involving sets of words that together form a lexical scale (e.g., <all, @xmath0. the use of an alternate excludes the other from the interpretation (e.g., some of the boys came +> (implicates) not all of the boys came). ## data given the scarcity of existing resources for gcis, we collected and annotated new data in nli format, focusing on different interpretations of the connective and. we assigned two different sets of labels, one in accordance with the pragmatic hypothesis (i.e., the implicature is labeled as an entailment) and the other with the logic hypothesis (i.e., only logical inferences are treated as entailments). methodology. to obtain data to test the temporal succession and the causal interpretation of and, we first used a multigenre english corpus (ukwac, @xcite ) to extract sentences where a main and a subordinate clause are explicitly encoded in a temporal succession or causal relation by a connective (e.g., frazier quit before i did). 2 then, we replaced the original connective with and (frazier quit and i did). the generated and the original sentences are, respectively, the premises and the hypotheses of our experiment (see the first two rows of from the premises used to test causal interpretation (e.g., he refused to sign and he lost his job) we produced new hypotheses where the clauses are linked by other temporal relations contradicting succession, namely precedence (before he refused to sign, he lost his job) and synchronous (while he refused to sign, he lost his job). this is to ensure that systems do not perform an enriched interpretation of and that goes in the wrong direction (either temporal or causal). since the pragmatic interpretation of and is temporal succession and this excludes a precedence or synchronous one, we assigned the gold pragmatic label contradiction to these pairs. we also wanted to test whether nli systems assign a logical interpretation to the connective and, namely commutativity. here we studied the influence of the semantics of the conjuncts: while commutativity is a more natural inference with conjuncts describing static situations (the rooms are comfortable and the food is super entails the food is super and the rooms are comfortable), with conjuncts describing dynamic situations it is less natural, since it is overridden by the inference stemming from pragmatic enrichment (he fell off a ladder and he had concussion contradicts he had concussion and he fell off a ladder). to obtain instances of inferences involving the commutativity of and with dynamic conjuncts, we used the sentences with a causal relation from our dataset. for instance, from the sentence he had concussion because he fell off a ladder we generated the premise he fell off a ladder and he had concussion and the hypothesis he had concussion and he fell off a ladder. for static conjuncts, we manually annotated clause pairs linked by and in ukwac, and selected only pairs where the main verb of both clauses is stative (the food is super and the rooms are comfortable) or has an habitual reading (platypus builds nest, and echidna develops pouch). while commutativity is entailed from the logic perspective, a contradiction would be produced if a pragmatic interpretation of and was selected, since temporal ## experiments systems. we used our data to evaluate a bert @xcite , a roberta @xcite and a deberta @xcite language model fine-tuned on mnli. for bert and roberta, we adopted the fine-tuned versions by @xcite . 3 we did not perform additional training, as our goal is to test existing systems and our dataset has been built only for evaluation purposes. results. we report in results results analysis. we first observe that the inferences drawn by the systems show inconsistent patterns. in many cases the systems assign a succession, precedence and synchronous interpretation to the same pair of conjuncts, which is an overt contradiction. second, the systems' behavior is not aligned with theoretical accounts of implicatures. linguistic theory predicts that only a limited set of relations between conjuncts can be inferred (among which succession and cause), while systems consider all the relations we tested as valid inferences. moreover, while the dynamic event type of the conjuncts is expected to lead to the rejection of the commutative interpretation in favor of an enriched one, systems prefer the commutative pattern irrespective of the context. ## conclusion we found that nli systems generalize "pragmatic" and "logical" inference patterns involving the connective and. this gives rise to unsatisfactory predictions, since in many cases inferences are not consistent with each other and are not aligned with human ones and theoretical accounts of implicatures. it should be noted that alternative accounts of implicatures exist: for scalar implicatures it has been shown that inference takes place with different strength depending on the context @xcite . a better assessment of the systems' abilities could be obtained by using implicature strength data. finally, at this stage we cannot draw general conclusions about whether our results also extend to systems trained on other nli datasets. based on the highlighted limitations of the tested systems, we argue for the need of a stronger convergence of neural systems with theories of gcis to improve systems' interpretation of and. @xcite proposed that i-implicatures can be explained by assuming that the hearer knows that the speaker tried to achieve her communicative goals by maximizing economy, and thus enriches the interpretation in stereotypical ways (since it assumes that the speaker has left stereotypical information unsaid). stereotypical relations between events in the form of event chains could be automatically collected from texts @xcite and provided as additional information to systems.