Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques (2023)

Abstract: This paper investigates the transferability of debiasing techniques across different languages within multilingual models. We examine the applicability of these techniques in English, French, German, and Dutch. Using multilingual BERT (mBERT), we demonstrate that cross-lingual transfer of debiasing techniques is not only feasible but also yields promising results. Surprisingly, our findings reveal no performance disadvantages when applying these techniques to non-English languages. Using translations of the CrowS-Pairs dataset, our analysis identifies SentenceDebias as the best technique across different languages, reducing bias in mBERT by an average of 13%. We also find that debiasing techniques with additional pretraining exhibit enhanced cross-lingual effectiveness for the languages included in the analyses, particularly in lower-resource languages. These novel insights contribute to a deeper understanding of bias mitigation in multilingual language models and provide practical guidance for debiasing techniques in different language contexts.

## introduction there has been a growing interest in addressing bias detection and mitigation in natural language processing (nlp) due to their societal implications. initially, research focused on debiasing word embeddings @xcite , but recent studies found that pretrained language models also capture social biases present in training data @xcite . hence, attention has shifted towards debiasing techniques that target sentence representations. these techniques include additional pretraining steps @xcite @xcite she was always nagging. he was always nagging. and projection-based methods that assume a bias direction @xcite @xcite . while debiasing techniques have been developed and evaluated for monolingual, and mostly english models, the effectiveness and transferability of these techniques to diverse languages within multilingual models remain largely unexplored @xcite . our research aims to bridge this gap by examining the potential of debiasing techniques applied to one language to effectively mitigate bias in other languages within multilingual large language models. we examine english (en), french (fr), german (de), and dutch (nl). figure 1 illustrates an example sentence pair included in the english crows-pairs dataset 1 , where the unmodified and modified parts are highlighted in blue and yellow respectively. it shows the predicted probabilities of the modified part occurring given the unmodified part across different debiasing languages. this study examines the cross-lingual transferability of debiasing techniques using mbert. mbert, trained on wikipedia data from diverse languages, possesses the capability to process and generate text in various linguistic contexts. despite balancing efforts, it still performs worse on low-resource languages @xcite . we investigate whether this performance disparity extends to gender, religious, and racial biases. related studies demonstrate the effectiveness of cross-lingual debiasing for individual techniques and selected bias scopes @xcite . we show how to reduce bias in mbert across different languages by conducting a benchmark of state-of-the-art (sota) debiasing techniques and providing guidance on its implementation. to facilitate further research and reproducibility, we make the code and additional data available to the research community foot_1 . our contributions can be summarized as follows: 1) we provide a benchmark of different sota debiasing techniques across multiple languages in a multilingual large language model. 2) we find that sentencedebias is the most effective for crosslingual debiasing, reducing the bias in mbert by 13%. 3) we provide implementation guidelines for debiasing multilingual models and highlight the differences in the cross-lingual transferability of different debiasing techniques. we find that most projection-based techniques applied to one language yield similar predictions across evaluation languages. we also recommend performing the techniques with an additional pretraining step on the lowest resource language within the multilingual model for optimal results. ## methodology this section introduces the data, debiasing techniques, and experimental setup respectively. ## results as shown in this phenomenon of overcompensation occurs in several underperforming techniques, as illustrated in is cross-lingual transferability of debiasing techniques possible? to shed some light on the insights that can be gathered from figure 2 , french is the worst-performing debiasing language for all evaluation languages except for dutch, where it is the best-performing one. we find that when evaluating in french, the gender bias is over- ## related work significant research focuses on the cross-lingual performance of mbert @xcite @xcite . limited research focuses on the cross-lingual transferability of debiasing techniques in mbert @xcite . @xcite use densray in english to debias chinese in mbert for gender. similarly, zhao et al. (2020) analyze the cross-lingual transfer of gender bias mitigation using one method. lauscher et al. (2021) also find that their proposed technique, adele, can transfer debiasing across six languages. other studies analyze biases contained in multilingual language models. kaneko et al. ( 2022 ) evaluate bias across multiple languages in masked language models using a new metric. @xcite study ethnic bias and its variability over languages proposing mbert as debiasing technique. finally, some studies also explore the cross-lingual transferability of downstream tasks @xcite . ## conclusion most studies focus on debiasing techniques for large language models, but rarely explore their cross-lingual transferability. therefore, we offer a benchmark for sota debiasing techniques on mbert across multiple languages (en, fr, de, nl) and show that debiasing is transferable across languages, yielding promising results. we provide guidance for cross-lingual debiasing, highlighting sendeb as the best-performing method, reducing bias in mbert by 13%. additionally, we find that, for the studied languages, debiasing with the lowest resource language is effective for techniques involving an additional training step (cda and do). this research is a first step into the cross-lingual transferability of debiasing techniques. further studies should include languages from different cultures and other multilingual large language models to assess generalizability.