Do Large Language Models Understand Conversational Implicature- A case study with a C hinese sitcom (2024)

Abstract: “Understanding the non-literal meaning of an utterance is critical for large language models(LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp,the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourcedfrom dialogues in the Chinese sitcom My Own Swordsman. It includes 200 carefully handcraftedquestions, all annotated on which Gricean maxims have been violated. We test eight close-sourceand open-source LLMs under two tasks: a multiple-choice question task and an implicature ex-planation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models,including GPT3.5 and several open-source models, demonstrate a lower accuracy ranging from20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation ofthe implicatures generated by LLMs on their reasonability, logic and fluency. While all mod-els generate largely fluent and self-consistent text, their explanations score low on reasonabilityexcept for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of theimplicatures in the conversation. Moreover, we find LLMs’ performance does not vary signif-icantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derivedfrom different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.”

## introduction the complexity of communication is largely epitomized by indirect, or non-literal utterances. a common instance is hinting at a busy schedule as a polite refusal to engage in an unwanted activity. how such implied meaning is understood in human communication has lone been a key subject of investigation in pragmatics research @xcite . evaluating the pragmatic understanding ability of large language models (llms) has drawn considerable attention in recent years as llms show remarkable ability for language understanding. recent studies have evaluated llms' pragmatic reasoning in multiple aspects, including scalar inference @xcite , discourse connectives @xcite , gradable adjectives @xcite and conversational implicatures @xcite . however, the above-mentioned evaluation are primarily in english, leaving a gap for pragmatic understanding in other languages. moreover, previous conversational reasoning datasets involve few turns of dialogue, while our daily communication usually involves much more context and turn taking. in addition, previous studies mostly employ binary-or multiple-choice questions @xcite , which is inadequate in the era of llm where it is possible to ask llms to generate explanatory text of the situation and directly evaluate its quality. question choices dialogue: 小郭： 知道这意味着什么吗？以后再 敢胡来，就不光是挨顿打的事 了。 薇： 烦死了，在家就是听爹娘罗嗦， 好不容易溜出来，倒听你罗嗦。 小郭： 什么？你是溜出来的？ 薇： 我说的那个溜，是溜冰的溜啊， 现在京城都在下雪，满地都是 冰，我是穿着冰刀，溜出来的， 嘿嘿…… english translation: xiaoguo: do you know what this implies? if you dare to mess around again, it won't just be about getting a beating. wei: so annoying. at home, i have to listen to my parents nagging. finally sneaking out, now i have to listen to you nagging. xiaoguo: what? you sneaked out? wei: when i said "sneak out," 1 i meant ice skating. it's snowing in the capital now, the ground is covered in ice. i wore ice skates and skated out, hehe... ## related work understanding non-literal meanings has long been considered a difficult task for language models. previous studies have explored the capabilities of language models in recognizing metaphors @xcite , humor @xcite and social commonsense @xcite . broad-scale datasets aimed at pragmatic reasoning collect data mainly through crowdsourcing or crawling from the internet @xcite . some others transform the existing tests for humans to train and evaluate language models @xcite . while the above-mentioned datasets have comprehensive coverage for pragmatic understanding, they are not ideal resources for evaluation of a specific type of pragmatic knowledge, that is, conversational implicatures derived by violation of the gricean maxims. in our work, we craft our dataset manually to make sure that each implicature can be classified to one or more gricean maxims. this enables us to perform a fine-grained analysis of llms' understanding of this particular pragmatic inference. previous attempts in evaluating llms' pragmatic understanding guided by the gricean maxims or cooperative principles do not have multi-turn dialogues from real-world situations. formalized under the theory of cooperative principles @xcite , @xcite automatically generated their dataset grice through a set of well-defined grammar rules. this grammar-based method ensures a good control for the pragmatic cues to be aligned with the gricean maxims. however, dialogues with fixed syntactic structures hardly represent the complexity of conversation in daily communication. in this study, we fill the gap by providing the multi-turn dialogue directly to the model. last but not least, there is a scarcity in chinese evaluation resources for pragmatic inferences. previous large-scale conversational datasets in chinese use text sources from sina weibo @xcite , douban conversational corpus @xcite and e-commerce dialogue corpus @xcite . however, all of them are not specifically aimed to test llms' pragmatic understanding. to address this gap, we select the chinese sitcom my own swordsman as the source to formulate pragmatic questions c c l 2 0 2 4 and construct a chinese dataset for evaluating llms' pragmatic reasoning ability. ## experiment 1: multiple-choice question for llms in this experiment, the models will see the dialogue and the four interpretations we manually created. the task is to choose the correct interpretation (i.e., the pragmatic one) of the utterance that contains an implicature. ## experiment 2: evaluating the quality of explanations generated by llms in the previous experiment, we ask llms to choose one answer from four choices. in this experiment, we design open-ended questions where the models are asked to generate explanations of the implicature, which will then be evaluated manually by native speakers of chinese, based on the reasonability/reasonableness, logic and fluency of the generated explanations. ## conclusion in this paper, we present swordsmanimp, the first fine-grained chinese dataset to evaluate llms' understanding of conversational implicature. in two experiments, we evaluate the state-of-the-art language models' pragmatic skills with two tasks. in experiment 1, we reveal that gpt-4 attains a human-level accuracy in answering multiple-choice questions, with other models lagging behind. we also find that no significant difference exists for both llms and human's accuracy with respect to conversational implicatures that violate different maxims. results from experiment 2 reflect the different patterns of generated text across llms and after human annotation of the quality of generated text from three dimensions, we reveal that while most models produce fluent text, they struggle to generate coherent and sensible explanations for the implicature, even if the model has achieved high accuracy in multiple-choice question.