Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models (2024)

Abstract: This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (e.g., ChatGPT). LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages. To systematically evaluate the cultural dominance issue, we build a benchmark of concrete (e.g., holidays and songs) and abstract (e.g., values and opinions) cultural objects. Empirical results show that the representative GPT models suffer from the culture dominance problem, where GPT-4 is the most affected while text-davinci-003 suffers the least from this problem. Our study emphasizes the need to critically examine cultural dominance and ethical considerations in their development and deployment. We show that two straightforward methods in model development (i.e., pretraining on more diverse data) and deployment (e.g., culture-aware prompting) can significantly mitigate the cultural dominance issue in LLMs.

## introduction large language models (llms) have become ubiquitous in various applications, such as machine translation @xcite , question answering @xcite , grammatical error correction @xcite and code intelligence tasks @xcite . however, these tasks usually consist of objective questions, whose answers can be determined as right or wrong. when it comes to subjective questions accompanied with no "standard" answers, we must pay attention to the "opinions" reflected by the llms. generally, these "opinions" can be shaped throughout the development of llms, from user-generated data collected on the internet, data combination during training, the ratio of responses related to english culture. the chatgpt's responses for non-english queries are more related to english culture than to the corresponding culture, demonstrating a predominance of english culture in chatgpt's outputs. and human alignment provided by crowd workers to the dedicated designs of model developers themselves @xcite . while there are pioneer works on revealing the "opinions" of llms @xcite , they are restricted to a single language (i.e., english) without considering the differences across languages. generally, for native speakers other than english, we expect llms to express "opinions" complying with the corresponding culture when asked for assistance. however, given the predominant use of english data in training sota llms (e.g., chatgpt), llms may inadvertently amplify dominant cultural narratives and further entrench existing cultural biases. as shown in figure 1 , chatgpt is dominated by english culture: inappropriate english-culture answers dominate the model output even when asked in non-english languages. such cultural dominance can lead to several negative effects, such as the loss of cultural diversity, promotion of stereotypes, increasing social and psychological inequality, and even violent conflict and economic impact @xcite . in this paper, we investigate llms' cultural dominance and call for developing more inclusive and culture-aware llms that respect and value the diversity of global cultures. notably, we focus on the potential negative effects of llms on "normal users," who are broader real-world users with no professional knowledge of prompt engineering. we construct a benchmark to comprehensively evaluate cultural dominance, considering both concrete (e.g., holidays and songs) and abstract (e.g., values and opinions) cultural objects. experimental results on the constructed benchmarks show that: while this paper focuses on the general-purpose interaction of llms for "normal" users across languages, the service provider can take necessary measures to enhance user experience by fostering cultural sensitivity. we show that two straightforward methods with different advantages can mitigate the cultural dominance problem: ## measuring cultural dominance to measure cultural dominance, we design a multilingual culture-relevant question set for concrete culture objects ( §2.1) and adopt two widely used multilingual value and opinion surveys for abstract culture objects ( §2.2). ## experiments we conduct experiments on the gpt family, including text-davinci-003, chatgpt, and gpt-4. we use the openai official playground to query text-davinci-003 and the official websites for chatgpt and gpt-4. we manually collect the responses from the webpage to mimic real-world usage scenarios. we also conduct repeated experiments with api to make the conclusions more reliable. specifically, we use the prompt to query gpt-4-1106 and gpt-3.5-turbo-1106 3 times with the default temperature of 0.8. more results are shown in the appendix, where our findings still hold. ## mitigation of cultural dominance while this paper focuses on llms' generalpurpose interaction with "normal" users across languages, the service provider can enhance the user experience by fostering cultural sensitivity. in this section, we present two simple and effective strategies for meeting the cultural requirements of a specific region. there are many possible ways to improve the localization of llm deployment. this paper does not aim to explore the whole space but simply to show that some reasonably straightforward implementations work well and that some methods (e.g., prompting) have almost no cost. ## conclusion this study exposes the cultural dominance of llms, particularly their tendency to reflect english culture even when queried in non-english languages. our experimental results on a constructed benchmark revealed that chatgpt is highly dominated by english culture. among the gpt family, text-davinci-003 is least affected by this issue, while gpt-4 is most affected. we propose two potential solutions to mitigate this problem: training llms on more diverse data, which can help reduce cultural dominance but at a higher computational and financial cost, and prompting llms by explicitly identifying the culture of the query language, a more cost-effective method that can improve performance on concrete cultural objects but is less effective on abstract ones. our findings underscore the need for developing more culture-aware llms that respect and value the diversity of global cultures. we hope that our research will encourage further exploration into this critical issue and inspire the creation of more culturally sensitive ai systems.