M ulti P rag E val: Multilingual Pragmatic Evaluation of Large Language Models (2024)

Abstract: As the capabilities of Large Language Models (LLMs) expand, it becomes increasingly important to evaluate them beyond basic knowledge assessment, focusing on higher-level language understanding. This study introduces MultiPragEval, the first multilingual pragmatic evaluation of LLMs, designed for English, German, Korean, and Chinese. Comprising 1200 question units categorized according to Grice’s Cooperative Principle and its four conversational maxims, MultiPragEval enables an in-depth assessment of LLMs’ contextual awareness and their ability to infer implied meanings. Our findings demonstrate that Claude3-Opus significantly outperforms other models in all tested languages, establishing a state-of-the-art in the field. Among open-source models, Solar-10.7B and Qwen1.5-14B emerge as strong competitors. By analyzing pragmatic inference, we provide valuable insights into the capabilities essential for advanced language comprehension in AI systems.

## introduction understanding a language involves not only the ability to process explicit information but also an awareness of the context that influences the meaning of each utterance @xcite . in human communication, context acts as a critical element as it provides a foundation upon which dialogue participants can understand and interact with each other more efficiently. with a shared context, communication becomes more facilitated, allowing subtle nuances to be successfully conveyed, which is essential for engaging in meaningful conversations @xcite . with recent advancements in generative ai, current llms have demonstrated capabilities that ex-aspect details utterance "there's the door." literal meaning a door is located over there. ## conclusion in this work, we present the first multilingual study of llms' capabilities of their pragmatic inference, particularly in the context of grice's theory of conversational implicature. our findings demonstrate the usefulness of multiprageval test suite in distinguishing the levels of comprehension among various proprietary and open-source models. the results reveal that among the models evaluated, claude3-opus and gpt-4 particularly stand out, with claude3-opus consistently outperforming gpt-4 by 6 to 10 points across all languages, affirming its state-of-the-art capability in pragmatic understanding. top-performing open-source models like solar-10.7b and qwen-14b demonstrate superior or comparable performance to lite-size proprietary models such as gpt-3.5, claude3-haiku, and mistral-small. the performance gaps across languages within models and individual grice's maxims further highlight language biases and areas for improvement. our findings, with the highest correlations with mmlu and math, suggest that general language understanding and complex logical reasoning are intricately linked to pragmatic inference abilities. this insight guides us towards further research to empirically demonstrate how these abilities relate to pragmatic reasoning.