Do Multilingual Large Language Models Mitigate Stereotype Bias? (2024)

Abstract: While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by systematically training six LLMs of identical size (2.6B parameters) and architecture: five monolingual models (English, German, French, Italian, and Spanish) and one multilingual model trained on an equal distribution of data across these languages, all using publicly available data. To ensure robust evaluation, standard bias benchmarks were automatically translated into the five target languages and verified for both translation quality and bias preservation by human annotators. Our results consistently demonstrate that multilingual training effectively mitigates bias. Moreover, we observe that multilingual models achieve not only lower bias but also superior prediction accuracy when compared to monolingual models with the same amount of training data, model architecture, and size.

## introduction with an increasing rate of adopting large language models (llms) in real-world applications such as healthcare, finance, and law @xcite , it is crucial to enhance their safe usage in terms of bias and fairness to avoid causing representational harm. many studies have investigated the biases encoded in llms, which vary across models and languages @xcite @xcite . figure 1 shows an example of bias using our english and multilingual models. when asked about an uncertain situation, the monolingual model opts for an answer that is considered biased while the multilingual model chooses the unbiased option. prior research has demonstrated the effectiveness of multilingual models in reducing language-specific ethnic bias @xcite . english only multilingual 'listen to this. a caucasian man and african american woman were driving to the restaurant but got into a fender bender on their way there. this resulted in a shouting match in the middle of the road.' the caucasian man i don't know figure 1 : an example from the bbq dataset @xcite , where a multilingual model shows an unbiased behavior compared to a monolingual model. furthermore, @xcite provided evidence that multilingual pre-training typically produces models with decreased bias. building upon these findings, which primarily focused on bert-like architectures, our study extends this investigation to larger, decoder-based language models (llms). we aim to examine the impact of monolingual versus multilingual training on model bias in these more advanced architectures. for this purpose, we train six novel 2.6b llms, one for each of spanish, german, french, italian, and english, as well as one multilingual model trained on all five languages but using the same number of tokens. we perform a human-validated automated translation of the crows-pairs @xcite and bbq @xcite bias evaluation benchmarks. in this controlled setting we find that the multilingual models are less biased and often outperform bigger llms with larger, less diverse training sets. ## related work much research has been done to analyze bias in the nlp community, a trend that has increased as the focus has moved toward deep and large models @xcite . the evaluation of bias in llms mostly focuses on models and benchmarks in the english language and culture @xcite @xcite . a survey of 146 papers found that in most studies there is no reasoning for why bias is harmful and to whom, which can lead to a mismatch between the objective and proposed methods @xcite . in this work, we use the definition from @xcite , also mentioned in @xcite , that stereotype bias in our experiments relates to representational harm that "occurs when systems reinforce the subordination of some groups along the lines of identity." metrics. there exists a broad range of metrics to quantify bias @xcite , and mitigation approaches to reduce it @xcite . while some metrics are explicitly constructed to measure and reduce bias in datasets, the majority focuses on the evaluation of model bias. @xcite differentiate between embedding-based bias metrics @xcite , probability-based bias metrics @xcite , and generated text-based bias metrics @xcite . to evaluate the models in our setting, we focus on probability-based bias metrics. ## approach to compare the encoded bias in mono-and multilingual models, first we use automatic translation to translate bbq @xcite and crows-pairs @xcite datasets and evaluate the translation quality with manual annotation. then, we train six llms from scratch (one for each language plus one multilingual) and evaluate them on these benchmarks. ## experiments we train monolingual and multilingual variants of our causal language models and evaluate them using a zero-shot setup on both the crows-pairs and bbq benchmarks and compare them with several recently developed llms. ## results and discussion results for the crows-pairs benchmark are shown in the heatmap in figure 2 . numbers shown are the percentage stereotype, we subtracted 50 from all the values, meaning that values greater than 0 indicate a tendency towards the stereotype sentence, while values less than 0 indicate a tendency towards the non-stereotype sentence. the perfect score is 0, where neither sentence is preferred over the other. we find that the multilingual model has scores that are closer to 0 in all languages compared to its monolingual counterpart and also opensource llms. breaking down the accuracy on the bbq dataset in figure 4 , we can also compare the accuracy of ambiguous and disambiguated contexts. we can observe that on the accuracy of ambiguous con-text, the multilingual model does much better than the monolingual models, while on the accuracy of disambiguated contexts, performance drops. the mixture of languages in the training data for the multilingual model seems to make it more conservative, hence the model is more likely to respond with "i don't know" when the information is insufficient, but this nature also causes loss of accuracy when dealing with the disambiguated samples (where the answer is always known). ## validity of the models to validate our model's capabilities beyond bias evaluation, we additionally conducted tests on the belebele benchmark @xcite , a common sense-based multiple-choice question-answering dataset designed to test the model's understanding capabilities in different language contexts. to fit our model, we also reformulated this dataset into qa format. the model's results are shown in from the belebele results, the monolingual models generally perform better than the multilingual model. this may be due to the fact that during the training of the multilingual model, the data for each language is only 20% of that for the corresponding monolingual model, leading to insufficient commonsense knowledge. however, given that our data-controlled models have less than half the parameters compared to other open-source models, our llm benchmark results are satisfactory. ## conclusion in this work, we systematically explored the relationship between the language of data a large language model is trained on and the stereotype bias that is encoded in the model. we trained six models with around 2.7b parameters from scratch using a causal language modeling objective and evaluated them on the crows-pairs and bbq benchmarks for english, french, german, italian, and spanish. to ensure that our approach can be extended to other languages and benchmarks, the datasets were automatically translated. for quality