Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation (2023)

Abstract: Gender bias is a significant issue in machine translation, leading to ongoing research efforts in developing bias mitigation techniques. However, most works focus on debiasing bilingual models without much consideration for multilingual systems. In this paper, we specifically target the gender bias issue of multilingual machine translation models for unambiguous cases where there is a single correct translation, and propose a bias mitigation method based on a novel approach. Specifically, we propose Gender-Aware Contrastive Learning, GACL, which encodes contextual gender information into the representations of non-explicit gender words. Our method is target language-agnostic and is applicable to pre-trained multilingual machine translation models via fine-tuning. Through multilingual evaluation, we show that our approach improves gender accuracy by a wide margin without hampering translation performance. We also observe that incorporated gender information transfers and benefits other target languages regarding gender accuracy. Finally, we demonstrate that our method is applicable and beneficial to models of various sizes.

## introduction in machine translation research, gender bias has emerged as a significant problem, with recent neural machine translation (nmt) systems exhibiting this bias @xcite . this bias can manifest in various ways, such as the misgendering of individuals based on stereotypes or defaulting to masculine gender translations. as a result, there is a growing need to address and mitigate gender bias in machine translation systems to ensure fair and unbiased translations that accurately reflect the intended meaning without perpetuating gender-based assumptions. le réceptionniste a salué l'avocat.... in this work, we investigate in detail the gender bias issue of multilingual nmt models. we focus on translating unambiguous cases where there is only one correct translation with respect to gender. we consider multiple target languages simultaneously with various gender-based metrics and find that even the state-of-the-art multilingual nmt systems still exhibit a tendency to prefer gender stereotypes in translation. therefore, we propose a new debiasing method for multilingual mt based on a new perspective of the problem. we hypothesize that the gender bias in unambiguous settings is due to the lack of gender information encoded into the nonexplicit gender words and devise a scheme to inject correct gender information into their latent embeddings. specifically, we develop gender-aware contrastive learning, gacl, which assigns gender pseudo-labels to text and encodes gender-specific information into encoder text representations. our method is agnostic to the target translation language as the learning is applied on the encoder side of the model and can be applied to debias pre-trained nmt models through finetuning. we also evaluate whether existing debiasing techniques for bilingual nmt are equally effective for multilingual systems and compare their effectiveness on different target languages. experimental results show that our method is highly effective at improving gender bias metrics for all 12 evaluated languages, with negligible impact on the actual translation performance. we find our approach applicable to various model architectures and very efficient in that it demonstrates significant gender accuracy improvement with just a few thousand steps of fine-tuning. we also discover that the debiasing effects extend to target language directions that are not trained on previous models. through further analysis, we demonstrate that our method effectively incorporates contextual gender information into the model encoder representations. in summary, the contributions of our work are as follows: ## method in this paper, we propose gacl, a gender-aware contrastive learning method for mitigating the unambiguous gender bias issue in multilingual machine translation. our approach is applicable to multilingual models that have encoder-decoder architectures and support english-to-many translation directions. we filter gender-related data and fine-tune the pre-trained nmt model with the filtered data. the overview of the method is shown in figure 2 . ## experimental framework in this section, we describe the details of the experiments, including the data, metrics, baseline methods, training architecture, and parameters. ## results in this section, we explore the gender bias issue of recent multilingual nmt models and report experimental results of the gender bias mitigation techniques of multilingual machine translation models. ## analysis in this section, we investigate how our gacl finetuning method affects the model representations regarding gender. specifically, we use 40 stereotypical occupation words from the winomt dataset, where 20 are stereotypically assumed male while the remaining 20 are assumed female. we label the "stereotypical gender" of an occupation word as defined by this gender assumption. we then construct a sentence with the template "he @xmath0." and "she @xmath1." as encoder input. here, the pronouns decide the "contextual gender" of the occupation word. finally, a single contextual representation vector is computed by taking the average of encoder output representations of the tokens that make up the occupation word. we use this same representation extraction process for both the baseline small-100 model and the small-100 model fine-tuned with gacl. @xcite to visualize the occupation representations in 2 dimensions, as shown in figure 5 . we observe that the representations for each occupation are clustered closely together regardless of the sentence context and model. this shows that the contextual gender has a relatively little contribution to the representation compared to the semantic meaning of the occupation. also, our fine-tuning method induces a relatively small change, preserving the semantic distinction between the occupations. finally, we note that the average distance between representations of different contexts is farther apart for the gacl representations (0.59) than the baseline representations (0.19), suggesting that the contrastive objective has guided the model to differentiate the occupation based on the gender context. ## conclusion in this work, we conducted an investigation into the gender bias of multilingual nmt models, specifically focusing on unambiguous cases and evaluating multiple target languages. our findings indicated that even state-of-the-art multilingual nmt systems tend to exhibit a preference for gender stereotypes in translations. we then proposed a novel debiasing method, gender-aware contrastive learning (gacl), which injects contextually consistent gender information into latent embeddings. our experiments demonstrated that gacl effectively improves gender accuracy and reduces gender performance gaps in multilingual nmt models, with positive effects extending to target languages not included in fine-tuning. these findings highlight the importance of addressing gender bias in machine translation and provide a promising approach to mitigate it in multilingual nmt systems.