Gender Lost In Translation: How Bridging The Gap Between Languages Affects Gender Bias in Zero-Shot Multilingual Translation (2023)

Abstract: Neural machine translation (NMT) models often suffer from gender biases that harm users and society at large. In this work, we explore how bridging the gap between languages for which parallel data is not available affects gender bias in multilingual NMT, specifically for zero-shot directions. We evaluate translation between grammatical gender languages which requires preserving the inherent gender information from the source in the target language. We study the effect of encouraging language-agnostic hidden representations on models’ ability to preserve gender and compare pivot-based and zero-shot translation regarding the influence of the bridge language (participating in all language pairs during training) on gender preservation. We find that language-agnostic representations mitigate zero-shot models’ masculine bias, and with increased levels of gender inflection in the bridge language, pivoting surpasses zero-shot translation regarding fairer gender preservation for speaker-related gender agreement.

## introduction with the rapid proliferation of intelligent systems, machine learning models reflecting patterns of discriminatory behavior found in the training data is a growing concern of practitioners and academics. neural machine translation (nmt) models have proven notoriously gender-biased, often result-ing in harmful gender stereotyping or an underrepresentation of the feminine gender in their outputs. in recent years, several approaches to debias nmt have been proposed, including debiasing the data before model training, the models during training, or post-processing their outputs. however, to the best of the authors' knowledge, it has yet to be explored how the phenomenon of not observing enough data, if any, to model language accurately affects gender discrimination in multilingual @xcite . to support translation between language pairs never seen during training (i.e., zero-shot directions), two widely-used approaches leverage the language resources (i.e., parallel data) available during training: pivot-based translation uses an intermediate pivot/bridge language (as in @xmath0, whereas zero-shot translation learns to bridge the gap between unseen language pairs using cross-lingual transfer learning. 1 in this work, we analyze gender bias in mnmt in the context of gender preservation, where gender information conveyed by the source language sentence needs to be preserved in the target language translation; in our experimental setting, source and target languages are grammatical gender languages that use a noun class system conforming with the gender binary, i.e., the classification of gender into the opposite forms of feminine and masculine, considered indicative of a person's biological sex. 2 we examine translations in terms of differences in gender preservation between both genders, which, if found, are evidence of gender-biased machine translation (mt). more precisely, we focus on the impact that bridging the gap between unseen language pairs has on the mt models' ability to preserve the feminine and masculine gender, unambiguously indicated by the source sentence, equally well in their outputs. our research questions are: rq1 how do zero-shot and pivot-based translation compare regarding gender-biased outputs for zero-shot directions? rq2 does the bridge language affect the gender biases perpetuated by zero-shot and pivotbased translations? rq3 do translation quality improvements of zero-shot models reduce their gender biases? the remainder of this paper is structured as follows. section 2 introduces the task of gender preservation in translation with relevant terminology and reviews related work on gender bias in nmt. section 3 describes our experimental design, tailored toward investigating cause-andeffect relationships of gender bias in mnmt. section 4 presents the data used and the evaluative procedure followed in our experiments. section 5 presents the experimental setup and results, and section 6 concludes with our summarized findings, limitations, and future research directions. ## terminology & related work in a large-scale analysis of the plethora of existing research addressing gender bias in nmt, @xcite categorize them based on two conceptualizations of the problem: research works focusing on the weight of prejudice and stereotypes in nmt, and studies assessing whether gender is preserved in translation. in this paper, we analyze gender bias in mnmt in the context of gender preservation, where for translation into a gendersensitive target language, the gender information conveyed by the source language needs to be retained in the target language translation. ## analyzing gender bias in mnmt in our experimental setting, information necessary to disambiguate gender was always conveyed by the source sentence (cf. figure 1a ) and, thus, available to the models. motivated by our research inquiry, we focused our investigation on the effect of bridging on gender preservation in mnmt between unseen language pairs, as illustrated broadly in figure 1b , exploring three influencing factors to learn about the cause-and-effect relationship of gender bias in mnmt: i) the approach taken to bridge unseen language pairs (i.e., using continuous representations for zero-shot translation or dis-crete pivot language representations); ii) the choice of bridge language; and iii) language-agnostic model hidden representations. zero-shot translation vs. pivoting: to bridge the gap between an unknown source-target language pair at inference, we took two different approaches using the same trained translation model. for pivot-based translation, we cascaded a model to perform @xmath1. as such, pivoting used the pivot language as an explicit bridge between the unknown language pair. for zero-shot translation, we used the same model to translate directly between the unknown language pair, relying on the model's learned semantic space where sentences with the same meaning are mapped to similar regions regardless of the language. compared to pivoting, zero-shot translation circumvents error propagation and reduces computation time, but achieving high-quality zero-shot translations is challenging. in light of our inquiry, we analyzed each approach's ability to preserve gender, comparing their performances for the feminine and the masculine gender. 4 bridge language: english often participates in most, if not all, language pairs in a training corpus, making english, a language limited to pronominal gender (with a few exceptions), the most reasonable choice for a bridge language. when translating into a genderless language (e.g., hungarian), the potential loss of gender information conveyed by the source sentence is unproblematic as it is evidently without detrimental consequence. however, when translating into a language with a higher gender-inflected system than english (e.g., french or italian), the loss of gender information poses a significant problem since the information necessary to disambiguate gender is virtually no longer existent (cf. bottom in figure 1b ). as preserving non-existent gender information is inherently impossible, also for humans, it is fair to assume that mt models have difficulty when encountering this phenomenon of gender ambiguity; the simplest solution is to resort to random guessing, with a 50% chance of choosing one gender over the other. any other gender distribution (̸ = 50:50%) is not reflective of random guessing but instead indicative of educated guessing based on knowledge or observations assumed to be true that can, however, include biases. against this background, we studied the role of the bridge language in gender preservation, focusing on the gender bias differences between pivotbased and zero-shot translation, using bridge languages with different gender-inflectional systems, including english (low gender inflection), german and spanish (high(er) gender inflection). german and english are both germanic languages. whereas in german, all noun classes require masculine, feminine, or neuter foot_4 inflection, english lacks a similar grammatical gender system. in german, the gender of the noun is reflected in determiners like articles, possessives, and demonstratives. on the other hand, spanish is a romance language with a binary grammatical gender system, differentiating masculine and feminine nouns; from a grammatical point of view, there are no gender-neutral nouns. the gender of nouns agrees with (some) determiners and, more often than in german, adjectives, making gender a pervasive feature in spanish. ## evaluation data & procedure for our evaluation, we built on the work of @xcite regarding the data and procedure used for our gender bias evaluation. ## experiments & results the code and scripts used for our experimental evaluation are available on github. foot_5 ## conclusion in this paper, we explored gender bias in mnmt in the context of gender preservation for zero-shot translation directions, i.e., unseen language pairs (french↔italian), compared the performances of pivoting and zero-shot translation using discrete and continuous representations respectively, studied the influence the bridge language has on both approaches, and examined the effect languageagnostic representations have on zero-shot models' gender biases. based on our experimental results, we addressed three research questions. rq1 how do zero-shot and pivot-based translation compare regarding gender-biased outputs for zero-shot directions? we find that zero-shot translation and pivoting achieve similar gender preservation performances, but zero-shot translation better preserves the feminine gender, which mitigates the masculine bias-the consistently worse feminine than masculine results across all evaluated models and both approaches-more than pivoting when bridging via english. rq2 does the bridge language affect the gender biases perpetuated by zero-shot and pivotbased translations? our experiments revealed that the bridge language affects gender biases in mnmt. for english, a language limited to pronominal gender (with a few exceptions), we find that zero-shot translation performs better than pivoting regarding a more fairly balanced preservation of feminine and masculine gender. using two richer gender-inflected bridge languages, spanish and german, revealed that with an increased level of gender inflection in the bridge language, pivoting surpasses zero-shot translation regarding fewer gender-biased outputs for utterances with speaker-related gender agreement. rq3 do translation quality improvements of zero-shot models reduce their gender biases? all three evaluated modifications encouraging language-agnostic hidden representations (cf. section 3) improved zero-shot models' ability to preserve the feminine and masculine gender and reduced the gap between better masculine and worse feminine results; they improved zero-shot models' performances to the point where they outperformed pivoting regarding more fairly balanced preservation of both genders when bridging via english. besides our findings, this work also features some limitations that can be addressed in future work. first, the data used in our experimental evaluation limited the scenarios to those examined. future work can examine the translation of sentences with mixed gender (i.e., sentences including feminine and masculine word forms) and directions, including languages from different language families and with different gender systems, to further study language differences. second, developing a large-scale gender-annotated corpus suitable for mnmt training could most likely be used to improve models' gender preservation performance. a well-performing gender classifier could be used to annotate the must-c dataset with token-or word-level gender labels. third, we believe that the metrics currently used to evaluate models' gender biases are not ideal. for instance, model outputs mismatching the reference translations used for evaluation are discarded, despite potentially being appropriate translations (e.g., synonyms); future work could explore using additional morphological analysis tools to include those translations in the gender bias evaluation. generally, inquiring about the phenomenon of gender bias in translation requires appropriate and established metrics; the lack thereof currently leaves room for improvement in evaluative procedures. while there is a lot of potential for further research on this topic, it is crucial to acknowledge that, ultimately, translation technology is bound by the principles of language, which subtly reproduces societal asymmetries and embeds signs of sexism, including masculine defaults and more subtle conventions by which expressions referring to females are grammatically more complex in many languages. consequently, combating gender biases in translation technology requires awareness of language use, as it is one of the most powerful means through which sexism and gender discrimination are perpetrated and reproduced.