Probing Pre-Trained Language Models for Cross-Cultural Differences in Values (2023)

Abstract: Language embeds information about social, cultural, and political values people hold. Prior work has explored potentially harmful social biases encoded in Pre-trained Language Models (PLMs). However, there has been no systematic study investigating how values embedded in these models vary across cultures. In this paper, we introduce probes to study which cross-cultural values are embedded in these models, and whether they align with existing theories and cross-cultural values surveys. We find that PLMs capture differences in values across cultures, but those only weakly align with established values surveys. We discuss implications of using mis-aligned models in cross-cultural settings, as well as ways of aligning PLMs with values surveys.

## introduction a person's identity, values and stances are often reflected in the linguistic choices one makes @xcite . this is why, when language models are trained on large text corpora, they not only learn to understand language, but also pick up on a variety of societal and cultural biases @xcite . while biases picked up by the plms have a potential to cause harm when used in a downstream application, they may also serve as tools which provide insights into understanding cultural phenomena. further, while studying ways of surfacing and mitigating potentially harmful biases is an active area of research, cultural biases and values picked up by plms remain understudied. here, we investigate cultural values and differences among them picked up by plms through their pre-training on web text. in a wide range of social science research fields, values are a crucial tool for understanding crosscultural differences. as defined by @xcite , values are the "core conceptions of the desirable within every individual and society", i.e., the foundation for the beliefs guiding a persons actions and on a society level the base for the guiding principles. we would like to highlight the difference we make between values and morals. the former, as conceptualised in this work, is concerned with fundamental beliefs an individual or a group holds towards socio-cultural topics, whereas the latter entails making a judgement towards individual or collective right or wrong. for a discussion around the intersection of morality and plms, we point the reader to @xcite . in this paper, we base our understanding of values across cultures on two studies: @xcite , which defines 6 dimensions to describe cross-cultural differences in values, and the world values survey (wvs) @xcite . both surveys provide numerical value scores for several categories on a population level across different countries and regions and are widely used to understand cross-cultural differences in values. plms are trained on large amounts of text from the web and have shown to pick up on semantic, syntactical, factual and other forms of knowledge which allow them to perform well across several natural language processing (nlp) tasks. since multilingual plms are trained on text in many languages, they have the potential to pick up cultural values through word associations expressed in those languages which are embedded in the pretraining texts. we therefore measure whether cultural values embedded in multilingual plms are correlated with the ones provided by the surveys. in wikipedia, which is one of the primary sources of training data for multilingual plms, cross-cultural differences have been established @xcite , and analysed by @xcite based on hofstede's theory. in this paper, we explore the novel research question of whether plms capture cultural differences in terms of values across different language models. we probe plms using questions from the values surveys of both hofstede's cultural dimensions the-ory and the world values survey. we reformulate the survey questions to probes and extract the answers to evaluate whether language models can capture cultural differences based on their training data. we focus on 13 languages, each of which is primarily geographically restricted to one country or region, to compare the results of the language models to the values surveys. the overall experimental setting for the paper is outlined in figure 1 . our work makes the following contributions 1 : ## related work expression and norms analysis of expression of identity and attitudes through language and its change has a long history in sociolinguistics @xcite . more recently, studies have used nlp to computationally analyse this change on social media data @xcite and link it to external factors like socioeconomic status @xcite and demographics @xcite . this has also been done to analyse broader societal trends like temporal change in attitudes towards sexuality (ch-wang and jurgens, 2021) and gender bias @xcite . further, there has been work on creating resources to analyse social norms and commonsense reasoning around them @xcite @xcite . hoover et al. (2020); roy et al. (2021) present work on extracting moral sentiment embedded in language using the moral foundation theory. to diversify visually grounded reasoning across different cultures, liu 1 the hofstede and world values survey probes used for our experiments can be found at https://github.com/copenlu/value-probing . et al. (2021) introduce a multimodal multilingual dataset. while there has been work on investigating and embedding social and moral norms, understanding values and their variation in a cross-cultural context remains understudied in the literature. @xcite provide a taxonomy of 54 values based on @xcite and provide a dataset and baselines for automatic value classification within the context of argument mining. the closest setup to ours would be one adopted by @xcite . they qualitatively assess the text generated by gpt-3, an autoregressive language model, by prompting it with english texts with a clear embedded value. they find that the embedded values in the generated texts were altered to be more in line with dominant values of us citizens, possibly due to its training data. our setup instead quantitatively measures whether cross-cultural differences in these values are preserved in multilingual language models when fed with the language spoken predominantly by people belonging to that culture. probing probing has been extensively used as tool to study a variety of knowledge and biases picked up by plms. this can be syntactic @xcite , semantic @xcite , numerical @xcite , relational @xcite or factual knowledge @xcite picked up by plms. probes can be created on both, at the word or sentence level @xcite . following work @xcite on studying gender bias in word embeddings, a number of studies have built on it to similarly probe for social biases embedded in plms @xcite @xcite @xcite . this can be done using cloze-style probing for measuring at an intra-sentence level @xcite or using pseduo-log likelihood @xcite based scoring @xcite . there are downsides to both approaches, the former potentially introduces unintended bias based on the tokens in the input probe while the latter assumes that all masked tokens are statistically independent @xcite . we choose the former since the probes in our case are carefully worded by social scientists with the explicit aim to extract bias towards a certain set of values. to the best of our knowledge, there is no existing work on probing for values embedded in plms in a comparative cultural context. in this paper, we explore how plms capture differences in values across cultures, and whether those differences reflect the ones found in values across cultures at large. to compare the plms' encodings of values, we compare them with established surveys capturing cross-cultual differences in values, namely hofstede's cultural dimensions theory and the world values survey (wvs) (section 4). we transform the survey questions introduced in those surveys for compatibility with plms by reformulating them semi-automatically to convert them into probes (section 5). then we translate these created probes from english into 13 geographically localised languages to conduct cultural value probing across 13 cultures (section 5). finally, we assess the variance in cross-cultural values embedded in plms and compare the probing results to the established values surveys in section 7. we investigate the following research questions as a first step to exploring this novel area of probing cross-cultural differences in values: rq1 do plms capture diversity across cultures for the established values? rq2 are there similarities in the embedded values across different plms? rq3 do values embedded in plms align with existing values surveys? 4 values surveys we base our work on previous studies on how values differ across cultures. as these are central to a number of research fields including political science, psychology, sociology, behavioral economics, cross-cultural research among others, there are a large number of studies utilising the scores provided by these values surveys. among the most common ones are hofstede's cultural dimensions theory and the world values survey. these studies build on the body of work in different fields: hofstede's theory is derived from management studies @xcite , while the wvs was developed in the field of political science @xcite . both studies have since been widely used across fields. ## probe generation in order to make the surveys compatible with language models, we reformulate the survey questions to cloze-style question probes @xcite that we can then perform masked language modelling inference on. since this is the task plms were trained on, we argue it is a suitable methodology to measure embedded cultural biases in these models. hofstede's cultural dimensions based on the english survey questions, the questions in the survey are manually reformulated to question probes (qps). this is done analogously to iterative categorisation, in which a set of possible labels (y + i , y - i ) corresponding to either end of the response options available in the survey are defined, which are the words the language models are probed for. the sentences are then reformulated to probes, and the labels masked. those labels are based on the answers of the original survey, for instance, the original question like have sufficient time for personal or home life with answer options consisting of different degrees of importance, the probe is reformulated to having sufficient time for personal or home life is @xcite ., where [mask] should be replaced by important or unimportant. ## discussion our experiments show that there are sizable differences in the cultural values picked up by the different multilingual models which are widely used for a number of language tasks, even when they are trained on data from the same source. this is in line with previous results @xcite and hints at the sensitivity of model design, training choices, and their downstream effect on model biases. while the values picked up by the models vary across cultures, the bias in the models is not in line with values outlined in existing large scale values surveys. this is an unexpected result since plms are known to pick up on biases present in language data that they are trained on @xcite . further, values are known to be expressed in language @xcite . hence, language models should pick up on and reflect cultural differences in values expressed in different languages based on their training text. a lack of such reflection points to possible shortcomings in representation learning when it comes to multilingual language models. there could be a number of reasons for this. one possible reason is the lack of diversity in multilingual training data. wikipedia articles in different languages are written by a small subset of editors that are not representative of the populations in those countries. further, large scale corpora like commoncrawl over-represent the voices of people with access to the internet, which in turn over-represents the values of people from those regions @xcite . such a bias being present in gpt-3 was explored by @xcite who show that lms trained on web text end up reflecting the biases of majority populations. other work also shows that pre-training text contains substantial amounts of toxic and undesirable content even after filtering @xcite . this highlights the need for including more diverse and carefully curated sources of data which are culturally sensitive and representative, in order for the models to better reflect the cultural values of those populations. in this pursuit, knowledge gathered from the field of cultural anthropology in following an emic methodology can prove to be useful @xcite . @xcite suggest that people express themselves differently online on twitter compared to survey responses. this is another potential reason for this mis-alignment. plms are used for a variety of different nlp tasks in different countries and hence to accommodate the usage of people from diverse backgrounds and cultures, it is not just important to have linguistic and typological diversity in training data, but also cultural diversity @xcite . having such a form of cultural knowledge is desirable for a number of real-world tasks including qa systems, dialogue systems, information retrieval. further, a lack of such faithful representation could lead to unintended consequences during the deployment of such models such as models imposing a set form of normative ethics over a diverse population that may not subscribe to it @xcite . this could also lead to models not being culturally sensitive and embedding harmful stereotypes @xcite . recently, work has been done on trying to align models with human values @xcite . while this may seem like a good idea at a first glance, also in light of the arguments presented above, some cultural values are harmful to portions of society, e.g. high levels of masculinity, which is connected to misogynistic language and perpetuating gender biases. thus, when working with cultural values, an auditing system @xcite with these value systems in mind and one that takes into account the downstream use case should be employed. ## conclusion in this study, we propose a methodology for probing of cultural values embedded in multilingual pre-trained language models and assessing differences among them. we measure alignment of these values amongst the models and with existing values surveys. we find that plms capture marked differences in values between cultures, though these in turn are only weakly correlated with values surveys. alongside training data, we discuss the impact training and modelling choices can have on cultural bias picked up by the models. we further discuss the importance of this alignment when developing models in a cross-cultural context and offer suggestions for more inclusive ways of diversifying training data to incorporate these values.