Are Multilingual LLM s Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings (2024)

Abstract: Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs “know” limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a “culture gap” in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticulturAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages.

## introduction large language models (llms) have achieved impressive results on question answering and reasoning tasks @xcite @xcite . however, when reasoning in situational context, human expectations may vary cross-culturally @xcite ., pragmatic failure, the inability to understand 'what is meant by what is said') and depend on the knowledge of the relevant cultural common ground (i.e., the shared knowledge based on which people within a culture reason and communicate, including concepts, common sense, etc. @xcite . understanding of such common ground in a cross-lingual setting is specif-question: what does the person mean by the proverb? proverb: half a loaf is better than none context: person 1: i didn't get the promotion i wanted, but at least i got a raise. person 2: of course, half a loaf is better than none. choices: a: a raise is better than nothing. b: a raise is worth nothing. answer: question: what does the person mean by the proverb? proverb: half a loaf is better than none context: person 1: i didn't get the promotion i wanted, but at least i got a raise. person 2: of course, half a loaf is better than none. choices: a: a raise is better than nothing. b: a raise is worth nothing. answer: question: what does the person mean by the proverb? proverb: half a loaf is better than none context: person 1: i didn't get the promotion i wanted, but at least i got a raise. person 2: of course, half a loaf is better than none. choices: a: a raise is better than nothing. b: a raise is worth nothing. ## related work prior work has evaluated the ability of llms to reasoning abstractly (ghosh and srivastava, 2022, recognize proverbs from short stories) or inference based on cultural norms @xcite in english and assessed the models' ability for matching proverbs across three languages (bigbench authors, 2023, with a small evaluation set). to the best of our knowledge, maps is the largest multilingual dataset that focuses on proverbs and sayings, with conversational contexts and an inference task. mabl @xcite is a task similar to ours but focuses on the multicultural understanding of novel metaphors and cross-lingual transfer. it is less suitable for studying memorization versus reasoning and does not study reasoning within a conversational context. @xcite and @xcite use conversational context to study pragmatic reasoning in english llms and the identification of parallels between humans and models, respectively. concurrently, @xcite proposed a culturallyaware natural language inference task based on cultural norms. however, they provide limited insights beyond english. while we also use conversational context, we focus on cultural common ground and multilingual aspects of mllms (with a larger dataset). other work on understanding the memory-retrieval mechanism in llms with english idioms @xcite , cultural knowledge @xcite @xcite or cultural value and bias @xcite @xcite . furthermore, we acknowledge existing work intended to study the formal and other types of reasoning in llms (such as the ones mentioned in @xcite , which are different in their goals from ours. ## maps -multicultural proverbs and sayings to help investigate our proposed research questions, we first present maps -a dataset of proverbs across six geographically and topologically diverse languages. maps consists of: (1) proverbs and sayings; (2) conversational usages as context; ## conclusion in this work, we use proverbs and sayings from different languages as an investigative tool to assess the ability of mllms to reason with cultural common ground. specifically, we study various mllms to evaluate their ability to memorize proverbs, reason with proverbs and sayings in different situational contexts and understand crosscultural communications using proverbs. to aid the investigation, we developed a multicultural proverbs and sayings dataset maps. our analysis shows that many models possess knowledge of proverbs and sayings, however, recognizing proverbs does not mean the model is able to reason with proverbs in contextual settings. indeed, we found that mt0 shows some culturallydiverse reasoning ability, but only to a very limited extent. we also found that the ability to reason in a zero-shot manner emerges with model scale, but the ability to understand a 'negative' question inversely correlates with the model scale. the disparities in culturally-diverse reasoning ability between languages grow with the model size, which raises concerns in terms of multilingual availability and points to the need for more robust mllms. finally, we defined and observed several culture gaps in cross-lingual communications. we hope to explore different aspects of cultural common ground in the future and to inspire novel work that facilitates inclusive cross-cultural understanding and communication with mllms.