LLM -Human Pipeline for Cultural Grounding of Conversations (2025)

Abstract: Conversations often adhere to well-understood social norms that vary across cultures. For example, while addressing parents by name is commonplace in the West, it is rare in most Asian cultures. Adherence or violation of such norms often dictates the tenor of conversations. Humans are able to navigate social situations requiring cultural awareness quite adeptly. However, it is a hard task for NLP models.In this paper, we tackle this problem by introducing a Cultural Context Schema for conversations. It comprises (1) conversational information such as emotions, dialogue acts, etc., and (2) cultural information such as social norms, violations, etc. We generate ~110k social norm and violation descriptions for ~23k conversations from Chinese culture using LLMs. We refine them using automated verification strategies which are evaluated against culturally aware human judgements. We organize these descriptions into meaningful structures we call Norm Concepts, using an interactive human-in-loop framework. We ground the norm concepts and the descriptions in conversations using symbolic annotation. Finally, we use the obtained dataset for downstream tasks such as emotion, sentiment, and dialogue act detection. We show that it significantly improves the empirical performance.

## introduction social norms define behavioral expectations shared across groups and societies @xcite . they can help explain the differences in the way people from different cultural backgrounds react to the same situation @xcite . @xcite describe norms of interaction as 'shared rules that implicate the belief system of a community', capturing the importance of representing norms when, either human or ai-systems, attempt to make sense of social interactions from different cultures. motivated by large language models emergent reasoning abilities @xcite @xcite several recent works attempted to create repositories of cultural norms using novel prompting approaches followed by automated verification of the generated descriptions @xcite @xcite . however, these efforts had limitations such as using synthetic conversations or focusing on a handful of situations. llms also tend to suffer from several reliability issues such as hallucinations @xcite or high sensitivity to prompt structure @xcite @xcite . these limitations motivate a more general and principled approach for cultural context understanding, which, we argue, should be viewed as a pragmatic reasoning task. norms are situated in specific settings and are associated with the social roles participants play @xcite . different expectations can be associated with individuals based on attributes such as their status, profession, or gender. furthermore, these expectations are situationdependent, for example, changing when engaging in professional or leisure activities. to that end, in this paper, we propose a novel cultural context grounding framework for conversations. we tackle three key problems. first, norm representation capturing the multi-party situated social expectations. second, norm induction, i.e., populating the suggested representation with norm concepts, emerging from conversational data, and associating them with relevant contextual information. finally, grounding the norm concepts in conversational data at scale and creating a dataset of norm-schema aligned conversations. our norm representation solution is inspired by the notion of scripts @xcite , i.e., structured representations of expected activities for different roles relevant to a specific scenario, in our case mapping to social scenarios. unlike past work, @xcite , that developed a schema representing expectations over situated actions, our goal is to capture how social norms manifest in conversational behavior. to allow for pragmatic inferences mapping the norm definition to conversations, we intentionally separate between factual components, capturing observed information about the settings and content of the conversation, and cultural norm components, capturing the expected behaviors and impact of violating that expectation. fig. 1(a ) depicts this separation. our framework follows a multi-stage approach to obtaining cultural information for existing conversations and grounding this information in conversation-specific details. we focus mainly on relevant social norms and their violations within these conversations. we then evaluate the usefulness of the created cultural context dataset both qualitatively and quantitatively. fig. 2 presents an overview of our framework. we describe it in §4. we leverage llms such as gpt-3.5 to generate a large but potentially noisy corpus of conversationspecific cultural information. then, we leverage an interactive human-in-the-loop process (alg. 1) that efficiently utilizes culturally proficient human annotation to organize this information into meaningful concepts. we further ground the generated descriptions in the conversation details, such as participants, their relationships, and their behavior using symbolic annotation. following this, we experiment with automated verification strategies to filter the obtained cultural information at scale. then, we organize the conversations and the obtained cultural information into a meaningful schema structure. finally, we propose a neural graph schema model that leverages obtained data to improve the empirical performance on conversation understanding tasks such as emotion detection, sentiment detection, and dialogue act detection across multiple datasets. overall, our contributions can be summarized as follows: ## cultural context grounding our high-level objective is to 'conceptualize & operationalize cultural understanding in conversational behavior.' we identify three key steps to make progress towards this objective: then, we introduce a robust pipeline for obtaining cultural information for real conversations and grounding the conversations in it. we efficiently leverage (1) native-culture human expertise, (2) llms as knowledge elicitors, (3) llms as symbolic annotators, and (4) llms as multi-agent verifiers, in this pipeline. we obtain a large-scale corpus for ∼23k chinese conversations from three existing datasets (tab. 1). we present an overview of the proposed pipeline in fig. 2 and a detailed pipeline description in §4.2. finally, we evaluate (1) correctness of the cultural information obtained using elicitor & symbolic annotator llms, and (2) usefulness of the schema for conversational understanding. we measure correctness against human annotations ( §5). we evaluate the usefulness of cultural schema data via conversational tasks such as emotion, sentiment, and dialogue act detection ( §6). we present full statistics of our pipeline data collection process in tab. 7 in appendix h. ## downstream task evaluation our experiments aim to evaluate the usefulness of cultural information and graph-based schema. we use empirical performance on conversational understanding tasks as a proxy for usefulness. we focus on 2 classes of models: no-context models and cultural context models. cultural context models are further divided into 2 types: models that consume cultural context as text and as a graph. we use 3 models for our experiments: roberta @xcite , llama-3.1 @xcite , and our convgraph model. we design a graph model that uses plm as a node encoder and leverages the schema structure in fig. 1(a ). we briefly discuss the datasets and tasks and then models used. datasets: we perform experiments on 3 tasks across 3 existing datasets. we conduct experiments with mpdd, cped, and ldc ccu chinese datasets ( §3). we use emotion detection (all 3 datasets), sentiment detection (cped), and dialogue act identification (ldc ccu, cped) tasks to benchmark the models. tasks: for emotion detection, mpdd is annotated using 7 emotion labels, cped with 13, and ldc ccu zh with 9 labels. dialogue act identification in cped is a 19 class task, and the ldc ccu chinese dataset uses 10 labels. cped sentiment is a 3-way annotation. we use the original train/validation/test splits for cped and mpdd datasets. for the ldc ccu chinese dataset, we use the ldc2022e18 release as the train set, the ldc2023e01 release as the validation set, and the ldc2023e20 release as the test set. in this dataset, neutral label for emotion detection and other for dialogue act identification are over-represented. we also find that several samples from these classes are missed annotations. hence, to avoid skewing the models, we down-sample these classes in all data splits to 1% of the actual labels. this makes these classes roughly the same size as the other classes. ## conclusion we propose a novel cultural context schema grounding pipeline. we introduce norm concepts which abstract over cultural beliefs. using the pipeline and llms, we create high-quality cultural context data and perform human evaluation. finally, we show that the dataset improves conversational understanding empirically.