N orm A d: A Framework for Measuring the Cultural Adaptability of Large Language Models (2025)

Abstract: To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs’ cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs’ performance ( \textless 82%) lags behind humans ( \textgreater 95%). In settings with abstract values and country information, model performance drops substantially ( \textless 60%), while human accuracy remains high ( \textgreater 90%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.

## introduction large language models (llms) have become globally widespread, engaging millions of users from diverse contexts and cultures @xcite . however, studies consistently highlight cultural biases in llm outputs, 2 particu-larly concerning the representation of various demographics @xcite , human values, and cultures @xcite . to be inclusive and effective across evolving cultures, llm outputs must embody pluralistic values and adapt to users' cultural nuances @xcite ; otherwise, there is a risk of providing disproportionate quality of service and fostering cultural alienation @xcite @xcite . previous work has largely focused on assessing knowledge and biases by probing llms with curated socio-cultural knowledge databases @xcite @xcite , often using direct questions about cultural norms, such as, "is it okay to eat with your left hand in india?". while these methods provide insights into what models know about different cultures, they do not fully evaluate their overall multi-cultural competence @xcite . we argue that true cultural competence requires models to not just possess cultural knowledge, but also to apply it flexibly to userspecific scenarios. @xcite highlights the benefit of cultural 'code-switching', which allows humans to adapt to different norms despite being attuned to their own cultural attributes. similarly, llms should be culturally adaptable @xcite , i.e., able to adjust their responses based on the user's cultural context. while it is still an open question as to how quickly or to what extent llms need to be adaptable, they can ensure effective communication across diverse scenarios by utilizing cultural values provided by or inferred from the user, rather than rigidly adhering to internal biases. to address the gap in evaluating the cultural adaptability in llms, we introduce the normad evaluation framework ( §3). using social norms as a proxy for culture @xcite , normad evaluates how models reason about the acceptability of social situations described in free- ## normad evaluation framework we introduce a multi-level evaluation framework to measure the cultural adaptability of llms, contrasting existing work that primarily measures knowledge ( §2.1). borrowing from @xcite , we say that an llm is culturally adaptable if its outputs are personalized or adapted towards multicultural users. 3 to be inclusive of diverse populations with varying values @xcite , we argue that a truly adaptable llm should perform well across diverse user-provided cultural contexts @xcite . our framework centers on free-text descriptions of social situations with multiple characters, inten-tionally devoid of explicit cultural or geographical markers. as shown in figure 1 , each scenario includes a social acceptability question about a character's actions. recognizing real-world scenarios' varying cultural information, we evaluate llms' adaptability across 3 levels of cultural specificity: rule-of-thumb (rot) detailed information necessary to answer social acceptability questions about character actions, simplifying the task to an entailment problem for both humans and models. for instance, figure 1 describes a situation where sarah is eating with their left hand and the rot is to "avoid eating with your left hand". ## normad-eti construction we demonstrate the utility of our framework by constructing normad-eti to explore llms' adaptability to social etiquette norms across different cultures. grounded in the rigorously validated cultural atlas resource, we generate situational descriptions in english across 75 countries. in this section, we describe our data construction pipeline (see figure 2 ): (1) social situation description, (2) automatic filtration, (3) human validation, and (4) verification of human performance. filtration validation value respect for personal hygiene and intercultural boundaries rule of thumb (rot) it is correct to use your right hand or both hands together when eating situation (yes/no/neutral) lisa reached out with her left hand to eat a cookie, is what lisa did acceptable? generate value, rot, story verify background ↔ rot rot↔ value country: india subcategory -indians may have dietary restrictions based on religious faith -the left hand is considered 'unclean' since it is the hand people generally use for washing themselves.... automatic filtering and regeneration verify situation relevant to rot inconsistent situations consistent generation 3x validation of gpt-4 filtration filtration judgment of humans and gpt-4 match final dataset validation rot ↔ cultural atlas value ↔ cultural atlas value ↔ rot situation ↔ rot label ↔ situation + rot 75 countries x 4 subcategories social-etiquette norms situation + country + value situation + rot human performance ## experimental setup we evaluate several language models on their ability to adapt to varying levels of cultural contexts. ## conclusion in this work, we introduce a novel hierarchical evaluation framework, normad, to assess the contextual adaptability of llms, a departure from most prior work which only probes cultural knowledge. instantiating this framework, we constructed normad-eti, a dataset of 2.6k social etiquette related situations spanning across 75 countries, evaluated for varying degrees of cultural norm specificity: specific country names, abstract highlevel values with country names, and finegrained rot. further, normad-eti involves four subcategories: 'basic etiquette', 'eating', 'visiting', and 'gifting', with three labels of adherence to social norms ('yes', 'no', 'neutral'). we find that models struggle across all levels of contexts, particularly with country +value and country setups, lagging significantly behind human performance. while larger models and kto optimization perform better, we see an increased performance skew across cultural zones, with english-speaking countries performing the best. models face significant challenges in the 'gifting' subcategory, which involves adhering to presentation, number, and color of gifts. further, they also exhibit inherent sycophancy biases, performing significantly better on situations conforming to social norms. these findings underscore the need for better contextualization, and more nuanced cultural adaptability in llms.