The Echoes of Multilinguality: Tracing Cultural Value Shifts during Language Model Fine-tuning (2024)

Abstract: Texts written in different languages reflect different culturally-dependent beliefs of their writers. Thus, we expect multilingual LMs (MLMs), that are jointly trained on a concatenation of text in multiple languages, to encode different cultural values for each language. Yet, as the ‘multilinguality’ of these LMs is driven by cross-lingual sharing, we also have reason to belief that cultural values bleed over from one language into another. This limits the use of MLMs in practice, as apart from being proficient in generating text in multiple languages, creating language technology that can serve a community also requires the output of LMs to be sensitive to their biases (Naous et al. 2023). Yet, little is known about how cultural values emerge and evolve in MLMs (Hershcovich et al. 2022). We are the first to study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning. Focusing on the fine-tuning stage allows us to study the interplay between value shifts when exposed to new linguistic experience from different data sources and languages. Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.

## introduction training lms on large text corpora has been shown to induce various types of (social) biases in multilingual lms (mlms) that affect which human values the model picks up on @xcite . however, human values vary per culture, which means that the cultural values that are reflected through their language (either explicitly or implicitly) will also differ. as mlms are trained on the concatenation of text from a wide variety of languages spoken in the world, we can expect different, and perhaps opposing, cultural values to be encoded in them simultaneously. this necessitates mlms to become inherently multicultural as well in order to appropriately serve culturally diverse communities @xcite . in fact, it has already been shown that mlms encode a distinct set of cultural values for different languages. however, those values do not tend to align with those collected from real human surveys conducted in the countries where the majority population speak the respective languages @xcite . as such, the multilingual nlp community is now faced with the new pressing challenge of better culturally aligning mlms to human values @xcite . thus, we aim to study how cultural values emerge and evolve in mlms to better understand and aid cross-cultural value alignment in the future. in particular, we hypothesise that training on multilingual data leads to an interaction of language-specific cultural values within the models, possibly steering a language's cultural bias into a direction that is unfaithful to the majority of that language's speakers. this raises an interesting set of questions on how languages exert influence on the encoding of cultural values. focusing on the fine-tuning stage, we study how cultural values are revised during training. for instance, given a set of fine-tuning languages, test languages, and data sources, when we fine-tune on a data source in a language a, and test in a language b, do we inadvertently induce the cultural values from a into b? and would the same effect be visible across all test languages or are the values encoded for some languages more prone to change? moreover, how much impact does the bias of the data source itself have versus the language used for fine-tuning? and can different data sources systematically enforce different alignments to human values? to better understand this cross-cultural interaction, we study the following questions: (q1) how do the fine-tuning language and data source affect the way in which cultural information is encoded and revised during fine-tuning? (q2) how do cultural value shifts change the alignment to human survey data? (q3) can we find patterns in the training examples that systematically influence how cultural values are revised? to this end, we conduct a set of controlled fine-tuning experiments using multi-parallel texts from data sources with neutral bias (wikipedia), religious bias (bible and quran) and political bias (news articles) across 4 fine-tuning languages and 13 test languages. we follow @xcite in using 200+ wvs survey questions to probe for cross-cultural values in pretrained and fine-tuned mlms. importantly, using survey questions as probes allows us to test the alignment between model predictions and human data. finally, we use a training data attribution (tda) to trace value shifts back to the data source. we find that, while fine-tuning language and domain source play a minor role in the revision of cultural information compared to the amount of fine-tuning data, fine-tuning languages can lead the cultural perspective of test languages into different directions. importantly, this can positively affect the models' alignment to human values. yet, overall, results vary considerably across test languages. moreover, our tda analysis provides interesting insights about the systematicity with which the model tends to rely on parallel data to instigate the same value shifts across languages. our results underpin the complexity of cross-language and cross-cultural interaction within mlms, and suggest that the semantic content of fine-tuning data might not be the main reason for value shifts. ## probing results as a baseline to our fine-tuning experiments, we first study the cultural profiles encoded in the pretrained models. in section 5.2, we then analyze how cultural information in the model changes as a result of cross-language and domain influence. ## correlation with human survey results in section 5.1, we studied which cultural values were encoded in the pretrained lm, and in section 5.2, how much and in which direction these could change after fine-tuning. we now test whether the changes we observed led the model to be steered into a direction that is better aligned to real human values. thus, we compute how much the spearman correlation between the ground truth profiles and the pretrained lm changed after finetuning. to select culturally diverse countries to test alignment to, we first compute cosine similarities between the ground truth profiles of our 13 test languages, and found that the profiles from germany and pakistan (0.84 similarity) and vietnam and serbia (0.88 similarity) deviated the most. ## tracing cultural value shifts we found that fine-tuning languages have similar effects across test languages. thus, as a complementary study, we test which training examples, and the languages they come from, influenced value shifts the most. we use trak, a tda method proposed by @xcite . we follow @xcite in treating the mlm objective as a multi-class classification problem, i.e., framing it as a sequence of v-way classification problems over masked tokens, where v is the vocabulary size. we use the trak library 3 for our implementation and project gradients down to 4096 dimensions, all other parameters are kept at default. see @xcite for a detailed explanation. per value shift we analyze the top 100 most influential training examples. are value shifts influenced by the same training examples across fine-tuning and test languages? in section 5.1, we saw that value shifts were mostly not unique to one fine-tuning language. thus, we test whether these shifts were actually influenced by the same training examples across fine-tuning languages. interestingly, we find that from the most influential examples in each fine-tuning language that instigate the same value shift, @xmath0. yet, when looking at the values that shift across test languages given the same finetuning language, we observe more consistency. for pbc, we find that across all language pairs, when fine-tuning in farsi and hindi, up to 20% of training examples are consistently relied upon across test languages, and for korean and russian 49 and 62% resp. these results suggest that the semantic content of fine-tuning data might not be the main reason behind the shifts. instead, the model tends to rely more on the same training examples within a fine-tuning language irrespective of test language. which languages instigate the value shifts during multilingual fine-tuning? we use the approach from @xcite to quantify cross-language influence by the average percentage of training examples that each fine-tuning language 3 https://github.com/madrylab/trak contributes to the most influential examples for each test language. in figure 6 , we see that for pbc, russian and farsi have on average the largest influence across test languages. interestingly, for tanzil, we instead see that russian and korean contribute the most. given that different trends across datasets could also be an artifact of randomness during fine-tuning, we repeat the experiment for pbc on a model fine-tuned with a different random seed, but confirm that the trends hold. while the large influence of russian could be explained by the fact that it has the second largest dataset for pretraining, the results indicate that the influence of languages on value shifts during multilingual fine-tuning is dependent on the content of the fine-tuning data. ## discussion and conclusion we studied to what extent fine-tuning languages and domain sources exert influence on cultural values encoded for a set of test languages in mlms. in particular, we tested how different fine-tuning setups can change the overall cultural biases across test languages differently, and in which cases this leads the model to be better aligned to human values. we found that fine-tuning language and domain source play a minor, but visible, role in the amount of value shifts compared to size of the finetuning dataset. moreover, results vary considerably across test languages. still, different fine-tuning languages can cause cultural profiles of test languages to be steered into different directions, which leads to varying effects on the models' alignment to human values. in addition, we find that multilingual fine-tuning better preserves the human cultural similarities between test languages within a lm. finally, our tda analysis shows that while different fine-tuning languages can lead to the same value shifts, the training examples that are relied upon vary. this suggests that the semantic content of fine-tuning data might no be the main reason for the shifts. instead, the model tends to rely on the same training examples within a fine-tuning language, and these examples have different effects on the manifestation of cultural values across test languages. hence, future work on value alignment likely requires a different adaptation approach for each test language. while multilingual nlp has made big strides in the past years, the field of crosscultural nlp is still in its infancy as many questions remain to be explored. we hope that our insights will inform future work on value alignment to enable more culturally-aware language technology.