CDE val: A Benchmark for Measuring the Cultural Dimensions of Large Language Models (2024)

Abstract: As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH principle, the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT-4’s automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.

## introduction large language models (llms), such as gpt-3.5, gpt-4 @xcite , and llama series @xcite have attracted widespread adoption from various fields due to their demonstrated human-like or even humansurpassing capabilities. to facilitate the development and continuous improvement of llms, various benchmarks have been used to evaluate llms' performance from different perspectives @xcite . for example, mmlu @xcite is used for assessing llms' multitask knowledge understanding, and covering a wide range of knowledge domains. @xcite figure 1: top: an example to illustrate different cultural orientations of people. bottom: the likelihood of cultural orientations of mainstream llms in three dimensions measured using cdeval. for instance, among the models evaluated, gpt-4 exhibits the lowest power distance index (pdi), whereas baichuan2 stands out with the highest pdi. to make llms better serve humans and eliminate potential risks, aligning them with humans has become a widely discussed topic @xcite . accordingly, there are several benchmarks for evaluating llms' human values alignment. @xcite introduced a benchmark comprising instances that are both helpful and harmless according to the hhh (helpfulness, honesty, and harmlessness) principle, a criterion that is widely accepted. @xcite proposed cvalues, a benchmark for evaluating chinese human values, with a focus on safety and responsibility. the above works primarily focus on aligning the llms with universal human values. however, human values are pluralistic @xcite , and individuals from different backgrounds often hold varied viewpoints on certain issues. for example, as illustrated in figure 1 (top), in terms of the cultural dimension of "individualism vs. collectivism (idv)", quotations from western contexts typically reflect an individualistic orientation, whereas those from eastern contexts tend to emphasize collectivism. therefore, llms should not only align with universal human values, demonstrating the capability to discern between right and wrong, but also honor and respect the rich tapestry of cultural diversity. motivated by this cultural diversity, we propose to investigate the cultural dimensions in llms. specifically, drawing from hofstede's theory of cultural dimensions @xcite , we identify and analyze six key cultural dimensions. figure 1 (bottom) showcases the results for three of these dimensions measured by our proposed llm culture benchmark. it is easy to observe that the llms also exhibit their inherent cultural orientations across different cultural dimensions. take "idv" as an example, gpt-4 exhibits a tendency towards individualism. in contrast, qwen-7b shows an inclination towards collectivism. as for "power distance index (pdi)", which measures the degree to which the members of a group or society accept the hierarchy of power and authority, we can find that gpt-4 leans towards equality but baichuan-13b shows a preference for hierarchy. we give more experiments in detail in section 4. in this paper, we first construct a benchmark for measuring the cultural dimensions of large language models, named cdeval. the construction pipeline is presented in figure 2 , which includes three steps. the first step is schema definition, which involves defining the taxonomy and the format of questions related to diverse culture dimensions. the second step is data generation using gpt-4, employing both zero-shot and fewshot prompts. the final step is checking the generated data manually under verification rules. the resultant dataset contains 2953 questions in total. an example question together with the options is illustrated in the bottom-right of figure 2 . the basic statistics of resultant benchmark are shown in cdeval is constructed by combining automatic generation with gpt-4 and human verification, and offers ease of testing, diversity, ample quantity, and high quality. ## the cdeval benchmark in this work, we employ llms as respondents, as discussed in @xcite , to investigate the culture of llms by administering questionnaires. this section details the development of constructing the questionnaire-based benchmark cdeval, and describes the evaluation process for llms' cultural dimensions. ## conclusion in this work, we introduce cdeval, a pioneering benchmark designed by combining automated generation and human verification to measure the cultural dimensions of llms. through comprehensive experiments across various cultural dimensions and domains, our findings reveal notable insights into the inherent cultural orientations of mainstream llms. the cdeval benchmark serves as a vital resource for future research, potentially guiding the development of more culturally aware and sensitive llms. in future work, it is crucial to explore how llms handle cross-cultural communication, particularly in understanding and interpreting context and metaphors from diverse cultural backgrounds. another vital area is investigating how llms manage conflicts arising from different cultural values, enhancing their capability for effective intercultural interaction.