Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on K orean Superstition (2025)

Abstract: As large language models (LLMs) become key advisors in various domains, their cultural sensitivity and reasoning skills are crucial in multicultural environments. We introduce Nunchi-Bench, a benchmark designed to evaluate LLMs’ cultural understanding, with a focus on Korean superstitions. The benchmark consists of 247 questions spanning 31 topics, assessing factual knowledge, culturally appropriate advice, and situational interpretation. We evaluate multilingual LLMs in both Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance. To systematically assess cultural reasoning, we propose a novel verification strategy with customized scoring metrics that capture the extent to which models recognize cultural nuances and respond appropriately. Our findings highlight significant challenges in LLMs’ cultural reasoning. While models generally recognize factual information, they struggle to apply it in practical scenarios. Furthermore, explicit cultural framing enhances performance more effectively than relying solely on the language of the prompt. To support further research, we publicly release Nunchi-Bench alongside a leaderboard.

## introduction the accessibility of large language models (llms) is expanding into everyday use beyond specialized domains such as legal and psychological counseling. however, their uneven cultural knowledge and contextual understanding are significant limitations @xcite , reducing their effectiveness in multicultural settings. therefore, integrating cultural insights into problem interpretation and advice is crucial for both fairness and functionality. previous research benchmarking llms' multicultural knowledge has typically verified factual 1 https://github.com/koreankiwi99/nunchi-bench information about specific cultures @xcite or explored the models' embedded values @xcite . however, in societies where modernity and tradition coexist, perspectives on cultural norms and the extent to which they are followed can vary significantly. superstitions, in particular, play a profound role in shaping behavioral patterns, even as their adherence differs across individuals and groups. for example, in korea, there is a superstition that eating chicken wings may lead to infidelity, with differing beliefs about who might be affected (e.g., only women, only men, etc.). instead of focusing on detailed cultural knowledge or the values of the models, our study aims to assess whether llms can interpret scenarios and develop superstition writing your name in red ink will bring death. ## conclusion this study introduced nunchi-bench, a benchmark for evaluating llms' cultural sensitivity and reasoning, with a focus on korean superstitions. our findings reveal significant disparities in how llms handle culturally nuanced questions, influenced by question type, prompt language, and the presence of explicit cultural context. to foster further research, we publicly release nunchi-bench and a leaderboard, encouraging ongoing improvements in cultural understanding. we also report updated results from recent models such as gpt-4.5 and claude 3.5 in appendix e, which show similar trends. future work should extend this benchmark to diverse cultural contexts, ensuring that ai systems are not only multilingual but also culturally adaptive.