Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys (2024)

Abstract: The cultural landscape of interactions with dialogue agents is a compelling yet relatively unexplored territory. It’s clear that various sociocultural aspects—from communication styles and beliefs to shared metaphors and knowledge—profoundly impact these interactions. To delve deeper into this dynamic, we introduce cuDialog, a first-of-its-kind benchmark for dialogue generation with a cultural lens. We also develop baseline models capable of extracting cultural attributes from dialogue exchanges, with the goal of enhancing the predictive accuracy and quality of dialogue agents. To effectively co-learn cultural understanding and multi-turn dialogue predictions, we propose to incorporate cultural dimensions with dialogue encoding features. Our experimental findings highlight that incorporating cultural value surveys boosts alignment with references and cultural markers, demonstrating its considerable influence on personalization and dialogue quality. To facilitate further exploration in this exciting domain, we publish our benchmark publicly accessible at https://github.com/yongcaoplus/cuDialog.

## introduction culture can be defined as the combinations of beliefs, norms, and customs among groups @xcite . implicit cultural cues hinted in dialogue utterances reveal different values and beliefs among speakers, which reflects their way of thinking @xcite and emotions @xcite @xcite . while pre-trained language models (plms) have shown impressive performance on dialogue tasks @xcite @xcite , their cultural bias in terms of values and their inconsistency in many other cultural aspects @xcite has severe implications on the prospect of employing them for interaction with speakers of diverse cultural backgrounds (hersh- encoder decoder (prediction) sen 1 dialogue tasks cultural survey sen 2 sen n sen 1 sen m culture enhancement classifier/ regressor cultures culture identification cultural vector china america german japan target fusion sen 2 ... ... @xcite as an additional vector for multi-turn dialogue culture identification and dialogue prediction enhancement, leveraging our proposed multicultural dialogue benchmark dataset, cudialog. previous studies in the field of cross-cultural nlp @xcite @xcite have primarily utilized probing methods to study the characteristics of models or agents. for instance, @xcite applied the hofstede culture survey @xcite to probe chatgpt, a prominent dialogue system, revealing a distinct disparity between the system and human society. this underscores the need to enhance dialogue agents' performance by incorporating cultural dimensions. however, developing culturally adaptive dialogue agents poses a significant challenge due to the scarcity of suitable datasets. while there are available multicultural corpora focused on specific domain tasks such as news @xcite and image captions @xcite , there is currently a lack of datasets specifically designed for cross-cultural dialogue tasks. to address this research gap, we introduce cu-dialog, an extensive english-language benchmark for multicultural dialogues. our benchmark covers 13 cultures and 5 genres, specifically designed to mitigate the impact of linguistic variations and emphasize implicit cultural cues. within cudialog, we propose two culture understanding tasks and one dialogue generation task, offering a comprehensive framework for evaluating and advancing cultural understanding in dialogue systems. specifically, as depicted in figure 1 , we design several baselines on culture classification and regression tasks, showing that cultural attributes behind dialogues can be identified. we leverage the soft cultural knowledge provided by the hofstede culture survey @xcite , which defines six cultural dimensions to measure the cultural attributes of different countries and provides statistical results for numerous nations. to utilize this external knowledge, we present a novel feature fusion mechanism based on an encoder-decoder generation framework, by considering using culture to assist separability in dialog generation. experimental results reveal that incorporating cultural value representation can improve alignment with references, indicating better cultural representation. in summary, our contributions are as follows: (1) we introduce cudialog, a multicultural dialogue benchmark dataset specifically tailored to different genres, enriched with cultural survey annotations. ## related work culture-oriented benchmarks. researchers have developed a range of culture-oriented benchmarks to investigate the impact of culture on language understanding and generation tasks. these benchmarks involve collecting and annotating multilingual and multicultural corpora to study cultural effects in downstream tasks. for instance, benchmarks have been introduced for news classification across different countries @xcite and for analyzing user statements reflecting different cultures using text and images @xcite . other benchmarks focus on detecting culture differences and user attributes, spanning both small-scale @xcite and large-scale @xcite datasets. furthermore, recent works have explored in-domain cross-cultural benchmarks, such as multilingual moral understanding and generation @xcite , and culture-specific time expression grounding @xcite . while zhang et al. (2022) cultural attributes learning. traditional approaches for capturing cultural differences often rely on probabilistic models, such as latent dirichlet allocation @xcite @xcite . however, the emergence of unsupervised learning and advancements in pre-trained language models (plms) have sparked interest in utilizing plms to learn cultural attributes and user profiles @xcite . culture-sensitive dialogue agents. previous studies @xcite have demonstrated the benefits of equipping dialogue agents with an understanding of cultural differences for natural language understanding (nlu) and generation (nlg) tasks, even in general natural language processing tasks. for example, @xcite proposed the use of a persona-specific memory network to jointly encode cultural background and user profiles, enhancing the nlg task for dialogue agents. @xcite introduced personalized approaches that respect individual beliefs expressed through user annotations. additionally, @xcite incorporated user queries, culturalrelated comments, and user profiles as encoded features to generate personalized responses, demonstrating the efficacy of leveraging both features in improving dialogue agent satisfaction. moreover, leveraging external knowledge by retrieving user-related cultural and attribute documents has shown promising improvements, providing additional guidance for model training @xcite . these works collectively highlight the value of incorporating cultural aspects into dialogue systems and leveraging personalized approaches for more effective and satisfactory interactions. while recent efforts have incorporated commonsense knowledge @xcite (a) mbert t-sne visualization for subtitles in 5 languages. ## cultural dimensions the hofstede culture survey @xcite identifies six cultural dimensions that capture different aspects of cultural values: power distance (pdi): reflects the acceptance of unequal power distribution within a society. individualism (idv): measures the level of interdependence versus self-definition within a culture. masculinity (mas): examines the emphasis on competition, achievement, and assertiveness versus caring for others and quality of life. uncertainty avoidance (uai): deals with response to ambiguity and minimizing uncertainty. long-term orientation (lto): describes how cultures balance tradition with future readiness. indulgence (ivr): focuses on the control of desires and impulses based on cultural upbringing. these dimensions offer valuable insights into the beliefs, behaviors, and attitudes that vary across societies. by incorporating these dimensions in our dataset for the corresponding countries, we provide a benchmark for evaluating the ability of dialogue systems to capture and adapt to cultural nuances. this enables researchers to assess the cultural sensitivity and adaptability of dialogue systems in a standardized manner. the survey results, freely available online for 111 countries, foot_0 serve as a valuable resource for integrating cultural dimensions into dialogue system enhancement and evaluation. yes yes yes no no no input: opensubtitles2018 english version sort countries, select top-13 sort genres, select top-5 filter meet length limit? split train/dev/test by movies reformat histories and predictions output: history, prediction, culture map with hofstede survey include in hofstede? figure 3 : the pipeline of the cudialog dataset construction process with our designed filtering strategy. ## multicultural dialogue dataset in this section, we introduce the collection, benchmarking, and statistics of our proposed multicultural dataset. the cudialog dataset contains four components: histories, golden predictions, culture label, and culture dimension scores, serving our proposed tasks, including culture classification, cultural alignment and dialogue generation, etc. data source. we gather multicultural dialogues from the opensubtitles 2018 dataset foot_1 @xcite , which comprises a vast collection of subtitles extracted from movies and television shows. the opensubtitles 2018 dataset offers extensive coverage of multiple languages, providing subtitle data in text format that is well-suited for training and evaluating a diverse range of nlp models. with its inclusion of various genres, such as action, drama, comedy, and documentaries, the dataset ensures an inclusive representation of linguistic styles and domains. while the dataset has been widely utilized in language identification @xcite , domain adaptation @xcite , and machine translation @xcite , it is essential to recognize that it also contains substantial cultural cues. to our knowledge, our work represents the first application of this dataset for culture-focused research, complemented by cultural annotations. language and culture selection. our research aims to explore the cultural differences underlying linguistic variations. we acknowledge that linguistic variations themselves serve as strong cultural features, which can have an impact on aspects such as common grounding and beliefs. to investigate the cultural cues related to beliefs and values, we conducted an analysis using a subset of 500 randomly extracted samples from the opensubtitles dataset. these samples were encoded by mbert and visualized using the t-sne method (van der @xcite , with a specific focus on the representation of data from five distinct countries. the visualization revealed distinct separations in the representation space based on different languages, making it challenging to capture cultural cues beyond linguistic variations. this motivated our decision to utilize english subtitles, as they exhibit less trivial separability (figure 2a ). as a result, our benchmark dataset universally employs english subtitles that encompass all cultures. the english subtitles in our dataset comprise both human-translated and machine-translated versions. furthermore, to establish a comprehensive benchmark dataset, we analyzed various genres and countries (as depicted in figure 2b ). we selected the top-five genres, namely action, comedy, drama, romance, and crime, as the basis for our dataset. in terms of country selection, we established a threshold of at least 50 movies per genre, ranked all countries accordingly, and chose the top-13 countries to represent cultures in our dataset. these countries include the usa, uk, france, japan, germany, canada, italy, south korea, india, spain, australia, china, and sweden. pipeline. our cudialog dataset construction pipeline (figure 3 ) involves gathering a comprehensive movie category index and extracting the corpus from each movie. we create multi-turn history: his mortal flesh belonged to the fire, his immortal soul to the flames of hell. a gag blocked his mouth. you'd have thought it was a corpse being led to its grave, "yet it was a living man whose torments were to gruesomely entertain the people." forgive me, i'll break off here. ## experiments 6.1 evaluated models to extensively evaluate the performance of currently available models, we select various models for evaluation, encompassing both encoder and encoder-decoder frameworks, as well as monolingual and multilingual models. specifically, we evaluate the following baselines for culture classification tasks: bert @xcite , multilingual bert, roberta @xcite , and xlm-roberta @xcite . for the culture regression task, we evaluate t5 @xcite , mt5 @xcite , bart @xcite , and mbart50 @xcite . for dialogue prediction, we evaluate mt5 on five genres. ## discussion in our investigation regarding culture identification, we strive to explore the extent to which models can effectively capture cultural attributes within the context of cudialog (rq1). additionally, we examine the integration of these identified cultural attributes into the demanding task of multi-turn dialogue prediction, thereby yielding outcomes that are both more satisfactory and diverse. this empirical analysis provides compelling evidence that incorporating cultural considerations can improve the performance of dialogue agents, thus validating the notion that cultural awareness plays a crucial role in enhancing their effectiveness (rq2). ## conclusion we introduced cudialog, utilizing opensubtitles 2018 for cultural identification and enhancing dialogue tasks. our approach goes beyond dialogue texts by introducing culture classification and regression tasks, capturing both coarse-grained and fine-grained cultural knowledge. by leveraging cues from cultural value surveys, we bridge the cultural nuances between dialogue agents and human society, resulting in effective dialogue prediction adaptation. further research in this area will advance the design of culturally aware dialogue systems that better meet user expectations.