C ulture S ynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis (2025)

Abstract: Cultural competence, defined as the ability to understand and adapt to multicultural contexts, is increasingly vital for large language models (LLMs) in global environments. While several cultural benchmarks exist to assess LLMs’ cultural competence, current evaluations suffer from fragmented taxonomies, domain specificity, and heavy reliance on manual data annotation. To address these limitations, we introduce CultureSynth, a novel framework comprising (1) a comprehensive hierarchical multilingual cultural taxonomy covering 12 primary and 130 secondary topics, and (2) a Retrieval-Augmented Generation (RAG)-based methodology leveraging factual knowledge to synthesize culturally relevant question-answer pairs. The CultureSynth-7 synthetic benchmark contains 19,360 entries and 4,149 manually verified entries across 7 languages. Evaluation of 14 prevalent LLMs of different sizes reveals clear performance stratification led by ChatGPT-4o-Latest and Qwen2.5-72B-Instruct. The results demonstrate that a 3B-parameter threshold is necessary for achieving basic cultural competence, models display varying architectural biases in knowledge processing, and significant geographic disparities exist across models. We believe that CultureSynth offers a scalable framework for developing culturally aware AI systems while reducing reliance on manual annotation.

## introduction cultural competence is a critical component of complex human emotional intelligence @xcite , encompassing key abilities such as empathy, cognition, and adaptability in understanding and navigating cultural differences @xcite . for individuals, cultural competence is not only demonstrated through awareness and understanding of cultural knowledge but also through the ability to adapt behavior and communication styles within multicultural contexts @xcite . as the utilization of large language models (llms) in interactions, communications @xcite , and workflows @xcite within global environments continue to grow, it is imperative to enhance the cultural competence of these models to optimize performance @xcite . this enhancement enables llms to more effectively interact with individuals from diverse cultural backgrounds worldwide @xcite . several benchmarks have been developed to evaluate the cultural competence of llms, primarily using multiple-choice and question-answer (qa) generation formats @xcite . multiple-choice benchmarks such as blend @xcite , @xcite , and mmlu @xcite gather culturally specific questions and manually annotate the correct and incorrect options. generation benchmarks like calmqa @xcite , na-tivqa @xcite , and culturebank @xcite compile naturally occurring questions from wikipedia and community web forums @xcite , in addition to human-written questions annotated by native speakers. other benchmarks, such as prism @xcite , cluster and filter culturally specific queries from real-world inquiries sourced via an english-speaking participant crowdwork platform. meanwhile, culturepark @xcite employs a different approach, creating a cultural benchmark through dialogues between llm-based multi-agent communication. however, existing cultural benchmarks encounter two primary challenges. first, cultural competence spans a wide range of domains (e.g., religion, social customs, law), yet most current benchmarks address only specific discrete cultural topics and lack a systematic taxonomy @xcite . this gap hinders comprehensive and systematic analyses of llms' cultural capabilities across varied domains. second, these benchmarks mainly rely on existing forums and manual annotation @xcite . while such benchmark construction methods ensure quality, they are resource-intensive, requiring substantial human labor and material costs, and cannot guarantee comprehensiveness. moreover, as the consumption of training data accelerates, llms increasingly face data shortages @xcite . consequently, employing automated methods to synthesize high-quality data becomes essential @xcite . to address the aforementioned challenges, we introduce culturesynth, as illustrated in figure 1 , a novel cultural framework featuring an automated data synthesis methodology. we propose the first hierarchically structured multilingual cultural taxonomy to tackle the issue of dispersed and unsystematic cultural topics. this framework integrates library classifications from five countries and regions to create a universal cultural taxonomy comprising 12 primary topics and 130 secondary topics (see the culturesynth-7 benchmark encompasses 7 languages (arabic ), totaling 19,360 qa pairs with thousands of cultural keywords per language. native speaker annotation of a representative subset validated the benchmark quality, achieving 95.8% question clarity, 83.5% cultural relevance, and 98.8% answer quality, with no safety concerns identified. using culturesynth, we conduct an extensive evaluation across 14 widely used language models, including gpt-4o @xcite , claude-3.5 @xcite , qwen2.5 @xcite , llama 3 @xcite , and the mistral @xcite series. the evaluation reveals a clear performance hierarchy among llms, i.e., chatgpt-4o-@xmath0.5-72b > claude-3.5-@xmath1. we identify a 3b-parameter threshold for basic cultural competence, below which models default to native-language functionality. model architecture plays a crucial role: mixture-of-experts architectures excel in cultural knowledge retrieval, while dense transformers perform better in long-context scenarios. our evaluation also uncovers distinct geographic and domainspecific biases, particularly gpt-4o's performance gaps in east asian contexts and claude-3.5's limitations in arabic and korean language processing. ## culturesynth this section presents the complete question-answer generation process of the culturesynth framework, as illustrated in figure 2 . ## conclusion in this paper, we present culturesynth, a framework for synthesizing culturally-aware qa pairs to evaluate llms' cultural competence. by combining a hierarchically cultural taxonomy with rag-based synthesis, we construct culturesynth-7, demonstrating high-quality synthetic data generation. our extensive evaluation, spanning 14 llms across 7 languages and 12 cultural topics, reveals that cultural competence depends on multiple factors beyond model scale, including training data composition, architectural design, and knowledge organization. we also identify different cultural understanding gaps, particularly in geographic and domain-specific contexts.