Pragmatic inference of scalar implicature by LLM s (2024)

Abstract: This study investigates how Large Language Models (LLMs), particularly BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), engage in pragmatic inference of scalar implicature, such as some. Two sets of experiments were conducted using cosine similarity and next sentence/token prediction as experimental methods. The results in experiment 1 showed that, both models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. In experiment 2, in which Question Under Discussion (QUD) was presented as a contextual cue, BERT showed consistent performance regardless of types of QUDs, while GPT-2 encountered processing difficulties since a certain type of QUD required pragmatic inference for implicature. The findings revealed that, in terms of theoretical approaches, BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model (Levinson, 2000). In contrast, GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model (Sperber and Wilson, 2002).

## introduction in recent years, there has been remarkable progress in natural language processing (nlp) thanks to the advent of transformers @xcite , from which numerous large language models (llms) have been developed. the effectiveness of these models relies on their ability to comprehend user input, which demands a focus on both semantics and pragmatics. semantics involves the literal meanings of words or sentences, while pragmatics focuses on context-dependent intended meanings. although advances in language modeling, particularly in neural vector representations like word2vec @xcite and glove @xcite , have shown significant progress in semantics, pragmatic inference has not received as much attention in nlp research, despite its importance for achieving increasingly natural conversations with users. pragmatic inference refers to the process of making inference by considering the contexts, intentions, and situations of language use. as a type of pragmatic inference, implicature is regarded as a linguistic phenomenon where the speaker conveys additional meaning or information that is not explicitly stated. one of the most commonly studied implicatures is scalar implicature, which indicates the quantity or range of a particular attribute, such as some. logically and semantically, the term some means at least one and possibly all. but, in actual language use, some is not always interpreted in this manner. pragmatically, some would lead the hearer to infer the meaning not all. for example, the sentence in (1) might be recognized as not all students passed the exam rather than at least one (or two in this case) and possibly all of them did. however, @xcite suggested that, in pragmatic discourse, whether some is interpreted semantically or pragmatically depends on the surrounding context, such as question under discussion (qud). qud refers to topics in a conversation that should be addressed with relevant responses at a later stage in communicative interaction @xcite @xcite . ## data collection to investigate the processing of scalar implicature by llms, we extracted sentences with 'some + np' structures from british national corpus (bnc) using nltk @xcite . among those, we collected sentences where 'some + np' was positioned as the subject due to the fact that the implicature generation is stronger when 'some + np' is positioned at the sentence-initial position compared to the sentence-final position @xcite ). in addition, we excluded sentences with multiple clauses to avoid the possibility of cancellation. finally, a total of 198 sentences were extracted and one example of the final data is presented as in ( 9 ). we refer to the sentences extracted through this process as some-sentences. both data and results of the experiments are publicly available. foot_0 ## experiment 1 previous experiments on human language processing have successfully captured pragmatic inference of scalar implicature even without context @xcite @xcite @xcite . likewise, experiment 1 aimed to investigate how llms interpret some-sentences without context, distinguishing between semantic entailment or pragmatic implicature. ## experiment 2 building on the findings from experiment 1, which showed that both bert and gpt-2 models prefer pragmatic interpretations to semantic interpretations in scalar implicature without context, experiment 2 aimed to explore whether the llms have more processing difficulties when implicature is required (i.e., upper-bound qud), compared to when implicature is not required (i.e., lower-bound qud). for this comparison, the context was manipulated using qud as a contextual cue. ## discussion through two sets of experiments, this study investigated how llms interpret scalar implicature, between semantic entailment and pragmatic implicature, in the absence of context and how qud, as a contextual cue, affects llms' processing of scalar implicature. experiment 1 investigated whether some-sentences in bertbase and gpt-2 exhibit greater similarity to semantic or pragmatic interpretations. the results showed that both models preferred the interpretation of pragmatic implicature over semantic entailment for some-sentences. experiment 2 aimed to investigate whether providing qud as a contextual cue would impact processing difficulties for bert-base and gpt-2, comparing between the upper-bound qud, where pragmatic implicature is required, and the lowerbound qud, where implicature is not required. as a result, bert showed no significant difference in processing difficulties based on quds, whereas gpt-2 showed more processing difficulties in the upper-bound qud. in conclusion, this study found that, only in a certain language model, gpt-2, greater processing difficulties were captured during pragmatic inference of scalar implicature, aligning with human language processing. bert and gpt-2, despite both being built on the transformer architecture, exhibited markedly different patterns regarding their theoretical approaches to the processing of scalar implicature. although both models shared the patterns of interpreting the term some as a pragmatic not all implicature rather than a semantic at least one and possibly all without context, bert exhibited no discernible difference in processing based on quds. this can be explained by default model where the meaning of some inherently defaults to not all @xcite . on the other hand, gpt-2 represented a clear difference in processing difficulties when manipulating context through the setting of qud, revealing that greater processing difficulties were captured in the processing of scalar implicature. this finding follows contextdriven model, consistent with the argument that not 8 in the study of @xcite , which investigated the effects of linguistic features on scalar implicature, they found that their model could make accurate predictions without considering the preceding context, while incorporating the preceding conversational context did not enhance and even diminished prediction accuracy. this led to the assumptions that only a context-independent utterance is sufficient and contextual cues may not be necessary for pragmatic inference, or that the model has not appropriately used contextual information. finding that context is unnecessary in scalar implicature may provide an explanation for our observation that bert in the experiment 2 showed no difference in processing efforts across quds. however, this explanation may not generalize to effectively capture the processing of scalar implicature in all llms, especially when taking into account the effects of qud on the processing of scalar implicature in gpt-2. liu et al. (2019) reported that features generated by pre-trained contextualizers were sufficient for achieving high performance across a broad range of tasks which explored the linguistic knowledge and transferability of contextualized word representations. however, they proposed that, for tasks requiring specific information not captured by contextual word representations, learning taskspecific contextual features plays a crucial role in encoding the requisite knowledge. within this framework, pragmatic implicature may either be pre-trained or require additional learning processes, depending on llms. therefore, it is crucial to recognize that different language models may incorporate diverse linguistic information and exhibit distinct processing patterns for the same linguistic phenomenon. furthermore, based on the argument of @xcite in which humans are influenced by context-driven expectations about unspoken alternatives, @xcite examined the bert model's variation in scalar implicature rate not just within a single scale @xmath0, @xmath1. this study revealed that the model's ability to make pragmatic inferences becomes stronger as more alternatives become available, which is depending on contextual predictability. this result leads us to expect that bert will show contrasting result if more alternatives are presented and the context becomes more predictable, despite the failure to make pragmatic inference within the provided context in the present study. in conclusion, the findings of this study suggested that llms are capable of pragmatic inference for scalar implicature without context. however, it is essential to understand the degree of contextual information utilization in each model and ensure appropriate learning for specific tasks. ## limitations while this study has advanced our comprehension of pragmatic inference in llms regarding scalar implicature, it faces limitations in three aspects. the first limitation is the absence of diverse constructions in which the scalar quantifier some appears. the exclusive use of experimental sentences featuring 'some + np' in the subject position within a single clause may not fully capture the broad spectrum of pragmatic interpretations that arise in various linguistic constructions and meanings in the real world. additionally, the number of data used in the experiments might be not large enough to generalize the findings. secondly, the study relies on only two of early transformer-based models, which may not reflect the performance of more advanced models that have emerged recently. since newer and more advanced models have demonstrated significantly higher performance across a wide range of tasks, using different models could yield varying results. lastly, in order to draw comparisons with human language processing, the experimental designs in this study deviate from conventional natural language inference (nli) tasks. moreover, the metrics used in this study (i.e., cosine similarity or next sentence prediction) may yield different results when other metrics are applied. the diversity in experimental methodologies can lead to variations in results, emphasizing the necessity for future research to take into account such differences. ## conclusion in this study, we discovered that llms interpret scalar implicature through pragmatic rather than semantic interpretation. additionally, the study identified the model that engage in pragmatic inference through the processing of scalar implicature using a contextual cue, such as qud, in contrast to the model that do not employ pragmatic inference. this study not only contributes to our comprehension of how llms process complex linguistic phenomena but also underscores the importance of considering pragmatics in nlp. by shedding light on the interplay between context and pragmatic inference, this study advances our understanding of llms and provides valuable insights for refining language models and applications in nlp.