W orld V alues B ench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models (2024)

Abstract: The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) → answer” from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely 11.1%, 25.0%, 72.2%, and 75.0% of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve <0.2 Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.

## value questions. the value questions in our wvb belong to 12 broad categories in the wvs, as mentioned in section 2.1. from each category, we include the first 3 questions as they appear in the wvs codebook in our wvb-probe. thus we use 36 value questions in total for this case study. ## demographic variables. we focus on the demographic variables of continent, residential area, and education level. for the continent variable, we consider 6 possible attributes: africa, asia, europe, north america, oceania, and south america. these are inferred from the country in which each survey is conducted, i.e., the b_country question in the wvs. the residential area variable corresponds to the h_urbrural question in the wvs and have two attributes: urban and rural. for the education level variable, we consider 4 attributes that are mapped from q275 in the wvs. isced 0 -1 are primary or no education, isced 2 is lower secondary, isced 3 -5 are upper to post secondary, and isced 6 -8 are tertiary education. there are thus 6 × 2 × 4 = 48 demographic groups according to the attribute combinations.