Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020)

Abstract: Multilingual representations embed words from many languages into a single semantic space such that words with similar meanings are close to each other regardless of the language. These embeddings have been widely used in various settings, such as cross-lingual transfer, where a natural language processing (NLP) model trained on one language is deployed to another language. While the cross-lingual transfer techniques are powerful, they carry gender bias from the source to target languages. In this paper, we study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications. We create a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations from both the intrinsic and extrinsic perspectives. Experimental results show that the magnitude of bias in the multilingual representations changes differently when we align the embeddings to different target spaces and that the alignment direction can also have an influence on the bias in transfer learning. We further provide recommendations for using the multilingual word representations for downstream tasks.

## introduction natural language processing (nlp) plays a vital role in applications used in our daily lives. despite the great performance inspired by the advanced machine learning techniques and large available datasets, there are potential societal biases embedded in these nlp tasks -where the systems learn inappropriate correlations between the final predictions and sensitive attributes such as gender and race. for example, @xcite and @xcite demonstrate that coreference resolution systems perform unequally on different gender groups. other studies show that such bias is exhibited in various components of the nlp systems, such as the training dataset @xcite , the embeddings @xcite @xcite as well as the pre-trained models @xcite . recent advances in nlp require large amounts of training data. such data may be available for resource-rich languages such as english, but they are typically absent for many other languages. multilingual word embeddings align the embeddings from various languages to the same shared embedding space which enables transfer learning by training the model in one language and adopting it for another one @xcite @xcite . previous work has proposed different methods to create multilingual word embeddings. one common way is to first train the monolingual word embeddings separately and then align them to the same space @xcite . while multiple efforts have focused on improving the models' performance on low-resource languages, less attention is given to understanding the bias in cross-lingual transfer learning settings. in this work, we aim to understand the bias in multilingual word embeddings. in contrast to existing literature that mostly focuses on english, we conduct analyses in multilingual settings. we argue that the bias in multilingual word embeddings can be very different from that in english. one reason is that each language has its own properties. for example, in english, most nouns do not have grammatical gender, while in spanish, all nouns do. second, when we do the alignment to get the multilingual word embeddings, the choice of target space may cause bias. third, when we do transfer learning based on multilingual word embeddings, the alignment methods, as well as the transfer procedure can potentially influence the bias in downstream tasks. our experiments confirm that bias exists in the multilingual embeddings and such bias also impacts the cross-lingual transfer learning tasks. we observe that the transfer model based on the multilingual word embeddings shows discrimination against genders. to discern such bias, we perform analysis from both the corpus and the embedding perspectives, showing that both contribute to the bias in transfer learning. our contributions are summarized as follows: gender bias in word representations word embeddings are widely used in different nlp applications. they represent words using low dimensional vectors. bolukbasi et al. (2016) find that, in the embedding space, occupation words such as "professor" and "nurse" show discrepancy concerning the genders. similarly, caliskan et al. (2017) also reveal the gender stereotypes in the english word embeddings based on the word embedding association test (weat). however, both works only consider english and cannot be directly adapted to other languages such as spanish. mc-curdy and serbetci (2017) reveal that bias exists in languages with grammatical gender while zhou et al. (2019) and lauscher and glavaš (2019) show that there is bias in bilingual word embeddings. however, none of them consider the cross-lingual transfer learning which is an important application of the multilingual word embeddings. to mitigate the bias in word embeddings, various approaches have been proposed @xcite . in contrast to these methods in english embedding space, we propose to mitigate the bias from the multilingual perspectives. comparing to @xcite , we show that a different choice of alignment target can help to reduce the bias in multilingual embeddings from both intrinsic and extrinsic perspectives. multilingual word embeddings and crosslingual transfer learning multilingual word embeddings represent words from different languages using the same embedding space which enables cross-lingual transfer learning @xcite . the model is trained on a labeled data rich language and adopted to another language where no or a small portion of labeled data is available @xcite . to get the multilingual word embeddings, @xcite learn a linear mapping between the source and target language. however, @xcite argue that there are some inconsistencies in directly learning the linear mapping. to solve those limitations, they constrain the embeddings to be normalized and enforce an orthogonal transformation. while those methods achieve reasonable results on benchmark datasets, they all suffer from the hubness problem which is solved by adding cross-domain similarity constraints @xcite . our work is based on the multilingual word embeddings achieved by @xcite . besides the commonly used multilingual word embeddings obtained by aligning all the embeddings to the english space, we also analyze the embeddings aligned to different target spaces. ## intrinsic bias quantification and mitigation in this section, we analyze the gender bias in multilingual word embeddings. due to the limitations of the available resources in other languages, we analyze the bias in english, spanish, german and french. however, our systematic evaluation approach can be easily extended to other languages. we first define an evaluation metric for quantifying gender bias in multilingual word embeddings. note that in this work, we focus on analyzing gender bias from the perspective of occupations. we then show that when we change the target alignment space, the bias in multilingual word embeddings also changes. such observations provide us a way to mitigate the bias in multilingual word embeddings -by choosing an appropriate target alignment space. ## extrinsic bias quantification and mitigation in addition to the intrinsic bias in multilingual word embeddings, we also analyze the downstream tasks, specifically in the cross-lingual transfer learning. one of the main challenges here is the absence of appropriate datasets. ## conclusion recently bias in embeddings has attracted much attention. however, most of the work only focuses on english corpora and little is known about the bias in multilingual embeddings. in this work, we build different metrics and datasets to analyze gender bias in the multilingual embeddings from both the intrinsic and extrinsic perspectives. we show that gender bias commonly exists across different languages and the alignment target for generating multilingual word embeddings also affects such bias. in practice, we can choose the embeddings aligned to a gender-rich language to reduce the bias. however, due to the limitation of available resources, this study is limited to the european languages. we hope this study can work as a foundation to motivate future research about the analysis and mitigation of bias in multilingual embeddings. we encourage researchers to look at languages with different grammatical gender (such as czech and slovak) and propose new methods to reduce the bias in multilingual embeddings as well as in crosslingual transfer learning.