Culturally-Aware Conversations: A Framework & Benchmark for LLM s (2025)

Abstract: Existing benchmarks that measure cultural adaptation in LLMs are misaligned with the actual challenges these models face when interacting with users from diverse cultural backgrounds. In this work, we introduce the first framework and benchmark designed to evaluate LLMs in realistic, multicultural conversational settings. Grounded in sociocultural theory, our framework formalizes how linguistic style — a key element of cultural communication — is shaped by situational, relational, and cultural context. We construct a benchmark dataset based on this framework, annotated by culturally diverse raters, and propose a new set of desiderata for cross-cultural evaluation in NLP: conversational framing, stylistic sensitivity, and subjective correctness. We evaluate today’s top LLMs on our benchmark and show that these models struggle with cultural adaptation in a conversational setting.

## introduction conversational llms are used for personal assistance, customer service, tutoring, therapy, etc., are increasingly deployed in global contexts. users who interact with these systems represent a rich set of nationalities, languages, and cultures, each with a distinct expectation of what constitutes a "good" interaction with an llm @xcite . to be effective across such diverse user groups, llms must be culturally aware, incorporating cultural context when conversing with users @xcite . a key component of cultural awareness in conversations is appropriate linguistic style 1 @xcite , which varies across cultures and additionally depends on setting, scenario, and social dynamics. however, most existing cultural benchmarks for llms are factual in nature and lack any focus on conversational dynamics @xcite . these benchmarks typically assess knowledge of cultural traditions, customs, or behaviors via trivia-style questions @xcite . while important, factual benchmarks do not generalize to the stylistic challenges of culturally sensitive communication. to evaluate llms in realistic, multicultural conversational settings, we propose the culturally-aware conversations (cac) framework & dataset designed for this task. our contributions are as follows: ## the cac dataset using our framework as the bedrock, we generate this dataset in three stages: scenario generation, conversation generation, and cultural matching. this pipeline is shown in figure 3 . 3 stage 1: generating scenarios. we begin by selecting a single situation and interpersonal relationship, as shown in figure 2 . next, we prompt openai's o3 model to generate a contextualized scenario using the situation and relationship. for example, the situation talking about personal accomplishments & failures and relationship friends yield the following scenario: over coffee, friend a tells friend b how failing an important exam pushed him to develop a more effective study routine. stage 2: generating conversations. we then prompt o3 to transform this scenario into a multiturn conversation. we first ask the model to generate a fixed first turn in the conversation: friend a: what changed for you after that exam? then, we ask o3 to generate a set of five responses that vary on the stylistic axis corresponding with the original situation. here are examples of the proud, neutral, and humble responses: all three of friend b's responses convey the same underlying message. however, the style of these responses varies along the pride-shame axis, evidenced by how much friend b brags about their new study routine. we generate one conversation per situationrelationship pair, for a total of 48 conversations and 240 possible responses. all 240 responses were validated by the authors to ensure that the stylistic range is properly reflected. during validation, minor edits were made to ∼30 responses to ensure they sounded natural and realistic. we show examples of generated scenarios and their corresponding conversations in table a2 . stage 3: cultural matching. upon generating conversations, we run a user study to understand which response is most appropriate in a given culture. we use a combination of volunteers from the authors' university and participants on prolific to recruit 24 annotators from eight countries -america, india, china, japan, korea, the netherlands, mexico, and nigeria. we then present each annotator with the conversations from the cac dataset consisting of (1) the fixed first turn, and (2) the set of five possible responses. annotators are asked to pick which response, depending on their personal set of ac-figure 3 : a depiction of how we use the cac framework to develop a contextualized conversation in our dataset. we walk through an example where the situation is giving critical feedback and the interpersonal relationship is boss-employee. in stage 1, we generate a specific scenario that reflects the situational and relational context. in stage 2, we use the scenario and stylistic axis to generate a conversation with a range of possible responses that vary on the given stylistic axis. in stage 3, we recruit annotators from a range of nations to determine which responses are most desirable in which cultures. ## evaluating today's top llms next, we evaluate how well today's llms understand the accepted stylistic ranges for interpersonal, professional, and day-to-day communication across cultures. we evaluate five models from openai, google, and anthropic by providing the situational, relational, and cultural context, and giving the first turn in the conversation and the five possible responses. we then ask the model to select the response that is most appropriate for that culture. to determine correctness, we check whether the predicted response falls within the culture-specific range of valid answers, after rounding for direct comparison. for example, if the accepted stylistic range is [1.25,2.67], then predictions of 1, 2, or 3 are considered correct. accuracy for each country is calculated as the proportion of correct predictions across all conversations. unsurprisingly, we find that llms perform best at adapting to western communication norms, with their highest accuracies observed for america and the netherlands. this imbalance is concerning because llm systems deployed in non-western contexts are less likely to align with local users' communication practices. ## conclusion the framework and dataset presented in this paper strive to bridge the gap between cultural psychology and generative ai. our work can be used to evaluate llms, inform conversational agents, and ultimately work towards models that are culturally competent and adaptive. this is especially important for building downstream systems, like chatbots, where context matters tremendously: the norms of appropriate communication differ sharply depending on whether a chatbot is deployed in a workplace, designed to tutor students, or intended to support individuals overcoming personal struggles. as a result, these systems need to adapt their understanding of social norms @xcite , implied language (havaldar et al., 2025a), and linguistic style @xcite . more broadly, llm systems that interact with diverse users operate not only within a cultural context but also within a situational and interpersonal context -the notion of "appropriate behavior" emerges from the interaction of all three. by formalizing these dimensions, our framework offers a path toward developing ai systems that better understand, respect, and adapt to diversity in communication.