MAKIE val: A Multilingual Automatic W i K idata-based Framework for Cultural Awareness Evaluation for LLM s (2025)

Abstract: Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness, often resulting in biased outputs. However, comprehensive multilingual evaluation remains challenging due to limited benchmarks and questionable translation quality. To better assess these disparities, we introduce MAKIEval, an automatic multilingual framework for evaluating cultural awareness in LLMs across languages, regions, and topics. MAKIEval evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. Leveraging Wikidata’s multilingual structure as a cross-lingual anchor, it automatically identifies cultural entities in model outputs and links them to structured knowledge, enabling scalable, language-agnostic evaluation without manual annotation or translation. We then introduce four metrics that capture complementary dimensions of cultural awareness: granularity, diversity, cultural specificity, and consensus across languages. We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems, across 13 languages, 19 countries and regions, and 6 culturally salient topics (e.g., food, clothing). Notably, we find that models tend to exhibit stronger cultural awareness in English, suggesting that English prompts more effectively activate culturally grounded knowledge. We publicly release our code and data.

## introduction the global deployment of llms in applications ranging from conversational agents to content creation makes their cultural awareness increasingly critical @xcite . without such awareness, models risk producing biased or culturally insensitive outputs, which can perpetuate stereotypes and harm user trust @xcite @xcite ). yet despite its importance, we ask fundamentally: how can we evaluate a text generation model's cultural awareness in a flexible and generalizable manner? figure 1 presents a realistic scenario: when prompted in different languages to describe a chinese person having dinner, the same llm produces outputs that differ in food entity specificity (i.e., whether the food items are culturally specific to china), diversity, and cross-lingual consistency. capturing these differences across languages in evaluation, especially under the flexible nature of text generation models, remains a major challenge. particularly, we identify three key limitations: i) test format: dominant evaluation methods often rely on simplified setups such as cloze tests, single-token prediction, or predefined cultural entity lists @xcite . while computationally efficient, these formats fail to capture how cultural knowledge naturally manifests in language generation @xcite . to address these limitations, we introduce makieval, an automatic approach for i) openended text evaluation, ii) leveraging wikidata as a language bridge and iii) enabling structured evaluation metrics and scalable, up-to-date evaluation. instead of static benchmarking, makieval is inspired from a human perspective and prior work: when humans describe daily life in their own region, they tend to be more detailed (high granularity), mention a wider range of items (high diversity), refer to culturally specific concepts (high culture specificity), and maintain consistent representations across languages (high cultural consensus) @xcite @xcite . for example, when asked about chinese breakfast, an english speaker unfamiliar with the culture might respond simply "a chinese eats noodles in the morning," whereas a chinese speaker would provide a richer description such as "youtiao (fried dough sticks) with soy milk, jianbing, or millet porridge in winter." this contrast highlights all four dimensions. these dimensions have long been discussed in psycholinguistics and linguistics, and they also resonate with recent nlp research on semantic abstraction, coverage, cultural awareness, and cross-lingual consistency. in what follows, we draw on this line of work to design four operational metrics that capture cultural awareness in multilingual generation. building on these metrics, we conduct comprehensive experiments to investigate how model origin, prompt language, and the mentioned country affect llms' cultural awareness. we found models align more with culturally related languages and their regions of origin. we publicly release our code and data. 1 our contributions are: some investigations examine abstract or highlevel aspects of culture, such as value systems @xcite @xcite , political ideology @xcite , gender norms @xcite , or expressions of stereotypes or hate @xcite . others focus on concrete, contentlevel evaluations, assessing how llms engage with specific cultural knowledge or practices @xcite @xcite @xcite . in this work, we also focus on the ideational branch, particularly on concrete cultural artifacts such as food, music, and clothing, which allow for a more structured, multilingual evaluation. ## makieval we propose makieval, an automatic and multilingual evaluation framework for evaluating cultural awareness. in figure 2 , we illustrate our pipeline, which consists of four steps: text generation, cultural entity extraction, wikidata-based entity match and metric-based analysis. ## evaluation metrics we propose four metrics to evaluate different aspects of cultural awareness in multilingual language models: granularity, diversity, culture specificity, and culture consensus. all metrics operate over a set of cultural entities extracted from the evaluated model's responses. we denote the union of all predicted entities across responses of a specific evaluated llm and a specific prompt as e. each entity @xmath0, from which we obtain relevant metadata (e.g., country of origin, entity type). the following sections define each metric in detail. ## results and analysis our evaluation is based on a total of 1,716 unique prompts, covering 13 languages, 6 cultural topics, 19 countries, and both contextualized and neutral settings. each prompt is used to generate 500 responses per model, resulting in over 85.8 million generated texts. makieval identifies a total of 24,042 matched cultural entities (details in appendix c.3), which form the basis for computing our four proposed metrics. in the remainder of this work, we explore the factors that influence cultural granularity, diversity, specificity and consensus, highlighting the complex interplay between model, language as well as cultural context. ## conclusion we present makieval, an automatic, scalable and multilingual framework for evaluating cul-tural awareness in llms. by combining promptcontrolled generation, structured entity extraction, wikidata grounding, and four culture metrics, our framework enables analysis of how models encode and express cultural knowledge across different languages, countries/regions, and topics. our results highlight that cultural awareness in llms is more complex than simple descriptions like a general bias to western culture. rather, it manifests in different forms, like the granularity, specificity and diversity of cultural entities. besides general model capabilities, we show that the prompting language plays a crucial role in evoking these cultural differences, even when no explicit cultural context is given. the sensitivity to cultural signals in english, the connection between countries and their local languages, and the cultural consensus that links regional areas emphasize the need for multilingual and multicontextual evaluation of cultural awareness. to foster further analysis of the cultural awareness of llms, we will release the full dataset of generated outputs and cultural entities for all model, language, country and topic combinations. our work also provides guidance for future research in identifying how these factors have been picked up by the models during training and alignment, and how culture is represented internally in the models. more broadly, our framework offers a generalizable paradigm for multilingual evaluation, our findings reveal systematic disparities that call for rethinking prompting and training beyond english, and our metrics provide actionable signals for improving fairness and inclusivity in global llm applications. we hope this work contributes to more equitable evaluation and deployment of language technologies worldwide.