Assessing Cross-Cultural Alignment between C hat GPT and Human Societies: An Empirical Study (2023)

Abstract: The recent release of ChatGPT has garnered widespread recognition for its exceptional ability to generate human-like conversations. Given its usage by users from various nations and its training on a vast multilingual corpus that includes diverse cultural and societal norms, it is crucial to evaluate its effectiveness in cultural adaptation. In this paper, we investigate the underlying cultural background of ChatGPT by analyzing its responses to questions designed to quantify human cultural differences. Our findings suggest that, when prompted with American context, ChatGPT exhibits a strong alignment with American culture, but it adapts less effectively to other cultural contexts. Furthermore, by using different prompts to probe the model, we show that English prompts reduce the variance in model responses, flattening out cultural differences and biasing them towards American culture. This study provides valuable insights into the cultural implications of ChatGPT and highlights the necessity of greater diversity and cultural awareness in language technologies.

## introduction the release of chatgpt by openai 1 in 2022 has sparked considerable attention and generated extensive discourse within both academic and industry spheres @xcite @xcite . after extensive training as the large language model gpt-3 @xcite , the official press release 2 reports that chatgpt has undergone fine-tuning through reinforcement learning with human feedback @xcite , resulting in its acquisition of unprecedented language and reasoning abilities and knowledge coverage. alongside its impressive proficiency in broad tasks @xcite , such as code generation, summarization, 1 https://chat.openai.com/chat 2 https://openai.com/blog/chatgpt for an average chinese, doing work that is interesting is ? "very important" or "of utmost importance". chatgpt, as a prominent dialogue agent, has been trained on a vast multilingual corpus that inherently embeds biases and cultural nuances @xcite . this necessitates a critical evaluation of the cultural values alignment between chatgpt and human society or deviates from them. although the biases of pretrained language models (plms) have been extensively studied @xcite @xcite , finding a consistent methodology for measuring their cultural alignment remains an open research question. in this paper, we propose a novel framework that leverages a wellknown social survey, i.e. hofstede culture survey @xcite , to probe the culture alignment between chatgpt and human societies. by examining various cultural norms and values present in different nations, we gain a deeper understanding of the challenges and limitations associated with accurately representing cultural differences in chat-gpt, as shown at the bottom of figure 1 . culture prompt 1 prompt 2 america (us) for an average american, [question #] is [options]. -china (cn) for an average chinese, [question #] is [options]. 对普通中国人来说，[问题 #] 是 [选项]。 germany (de) for an average german, [question #] is [options]. for an average japanese, [question #] is @xcite . ## related work cultural differences in nlp. culture can be defined as the pattern of thinking, feeling and reacting, distinguishing human groups @xcite . @xcite propose four dimensions of culture relevant for nlp, including linguistic form and style, common ground, aboutness, and values. @xcite construct a cultural background prediction benchmark, focusing on different expressions across countries. @xcite propose an imagenet-style benchmark to evaluate visual reasoning across different cultures. recent studies @xcite @xcite emphasize the impact of social bias in training data on nlp models, claiming they widen existing inequality gaps-also across cultures. ## method in this section, we elaborate on the survey, foot_0 our probing prompts, and interaction strategy below. ## conclusions based on our designed probing pipeline and prompts towards hofstede culture survey within five cultures, we have assessed the cultural alignment and consistency of results generated by chat-gpt as a representative dialogue agent. our analysis reveals that chatgpt can be better aligned with american culture, likely due to the abundance of english training corpus. however, we have also identified a significant gap in cultural adaptation between chatgpt and human society in our investigated questions. moving forward, future work in cultural alignment could focus on promoting cultural response consistency, enhancing cultural generalization and cultural adaptation.