Mitigating Gender Bias in Machine Translation with Target Gender Annotations (2020)

Abstract: When translating “The secretary asked for details.” to a language with grammatical gender, it might be necessary to determine the gender of the subject “secretary”. If the sentence does not contain the necessary information, it is not always possible to disambiguate. In such cases, machine translation systems select the most common translation option, which often corresponds to the stereotypical translations, thus potentially exacerbating prejudice and marginalisation of certain groups and people. We argue that the information necessary for an adequate translation can not always be deduced from the sentence being translated or even might depend on external knowledge. Therefore, in this work, we propose to decouple the task of acquiring the necessary information from the task of learning to translate correctly when such information is available. To that end, we present a method for training machine translation systems to use word-level annotations containing information about subject’s gender. To prepare training data, we annotate regular source language words with grammatical gender information of the corresponding target language words. Using such data to train machine translation systems reduces their reliance on gender stereotypes when information about the subject’s gender is available. Our experiments on five language pairs show that this allows improving accuracy on the WinoMT test set by up to 25.8 percentage points.

## introduction most modern natural language processing (nlp) systems learn from natural language data. findings of social sciences and corpus linguistics, however, indicate various forms of bias in the way humans *first authors with equal contribution. use language @xcite @xcite . thus the resulting nlp resources and systems also suffer from the same socially constructed biases, as well as inaccuracies and incompleteness @xcite @xcite @xcite @xcite . due to the prevalent use of nlp systems, their susceptibility to social biases becomes an increasingly significant concern as nlp systems not only reflect the biases learned but also amplify and perpetuate them further @xcite @xcite . this work concerns mitigating the manifestations of gender bias in the outputs of neural machine translation (nmt) systems in scenarios where the source language does not encode the information about gender that is required in the target language. an example is the translation of the english sentence "the secretary asked for details." into latvian. in english, the gender of "secretary" is ambiguous. in latvian, however, there is a choice between the masculine noun "sekretārs" and the feminine noun "sekretāre". in cases when sentences do not contain the necessary information, nmt systems opt for translations which they have seen in training data most frequently. acquiring the necessary information, however, might require analysis of the text beyond the level of individual sentences or require incorporation of external knowledge. falling back to biases, however, happens not only in the absence of the required information as nmt systems produce stereotyped translations even when clues about the subject's correct gender are present in the sentence @xcite . this is in line with findings by @xcite who suggest that nmt systems produce biased outputs not only because of the biases present in data but also due to their tendency to exacerbate them. to provide means for incorporation of external and explicit gender information, we propose a method for training nmt systems to use wordlevel gender annotations. to prepare training data, we project grammatical gender information of regular target language words onto the corresponding source language words. albeit in some cases redundant, we expect that the grammatical gender information contains a useful learning signal that helps narrowing down the lexical choice of the correct target translation. as a result, the nmt system learns to rely on these annotations when and where they are available. in particular, in experiments on five language pairs, we show that the methods proposed here can be used in tandem with off-the-shelf co-reference resolution tools to improve accuracy on the winomt challenge set @xcite by up to 25.8 percentage points. ## methods when translating from languages without grammatical gender to languages with grammatical gender, sākot ar polijas santehniķi un beidzot ar indijas datorprogrammētāju vai grāmatvedi ! out with the polish plumber , in with the indian computer programmer or accountant ! u u f m u u u f m u f u target to source alignment target sentence gender annotations 1-0 1-1 1-2 2-3 3-4 6-6 6-7 6-8 7-9 8-10 8-11 9-12 10-13 11-14 gender annotations certain words in the source sentence may not contain all the necessary information to produce an adequate and accurate translation. examples are pronouns (e.g. i, me, they, them, themselves), animate nouns such as job titles and proper nouns such as names and surnames, which depending on the sentence context can be ambiguous and consequently can be translated poorly. previous work has also shown that nmt systems are better at translating sentences that align with socially constructed gender stereotypes because they are more frequently seen in training data @xcite . to circumvent the degradation of nmt outputs due to 1) socially constructed biases and 2) absence of necessary information, we propose a method for training nmt systems to be aware of and use wordlevel target gender annotations (tga). for training, we use data where regular source language words are annotated with the grammatical gender of their target language translations. we obtain such data by, first, morphologically tagging target language sentences to obtain information about their grammatical gender-f for feminine, m for masculine, n for neuter, and u for cases where grammatical gender is unavailable. then, we use word-level statistical alignments to project this information from the target language to the source language words (see figure 1 for an illustration). we use source-side factors @xcite to integrate the projected annotations as an additional input stream of the nmt system. to ensure that the nmt systems are capable of producing adequate translations when gender annotations are not available-a frequently expected case at the test time-we apply tga dropout. we do so by randomly replacing annotations for a random number of words with u. while useful for animate nouns, such annotations might seem otherwise redundant because the majority of nouns in training data can be expected to be inanimate. however, for some inanimate nouns, the target language grammatical gender annotations can help narrowing down the lexical choice during training. an example is the translation of "injury" into latvian, where "injury|f" would result in "trauma" while "injury|m" would correspond to "ievainojums". besides disambiguating animate nouns, annotations also disambiguate the grammatical gender of pronouns, proper nouns. furthermore, grammatical gender annotations also concern adjectives and verbs, which in some languages have to agree in gender with the nouns they describe. consequently, we expect that during training the nmt model will learn to use these annotations, as they contain valuable information about words in the target sentence. at inference time, we lean heavily on the observation that there the grammatical gender of animate nouns, pronouns, and proper nouns, and the intended referential gender coincide considerably. this is, however, a heuristic and not a rule (see @xcite for counterexamples). nevertheless, we assume that it is possible to use tga in a referential sense of gender, thus injecting the nmt model with additional information about the subject's gender. sources of such information can vary; in this paper, we showcase how to use tga together with off-the-shelf co-reference resolution tools. ## experimental setting languages and data in all our experiments, we choose one source language without grammatical gender and five indo-european languages in which nouns have grammatical gender (see ## results and discussion results from experiments evaluating gender bias using the winomt test suite are provided in results from experiments using tga with oracle gender information show an improvement in winomt accuracy and ∆g for all language pairs (see effect on bleu as expected, using tga with reference sentence grammatical gender annotations has a positive effect on bleu, thus confirming our hypothesis why and how the nmt system learns to rely on tga as an additional source of information during training (see ## conclusions we proposed a method for training mt systems to use word-level annotations containing information about the subject's gender. to prepare training data, the method requires a morphological tagger to annotate regular source language words with grammatical gender information of the corresponding target language words. during inference, annotations can be used to provide information about subjects' referential or social gender obtained by analyzing text beyond sentence boundaries or externally. in experiments with five language pairs, we showed that using such gender annotations reduces nmt systems' reliance on gender stereotypes in principle. we then further showed one way for how these findings can be used in practice by using off-the-shelf co-reference resolution tools. the method proposed here decouples the task of acquiring the necessary gender information from the task of learning to translate correctly when such information is available. thus system's ability to use such information can be achieved independently from its availability at training time. this allows for application-specific sources of gender information. examples are the translation of chat or social media content, where users may choose to indicate their gender or translation of whole documents, where gender information may be obtained using annotations and anaphora resolution. thus, we believe that the methods proposed here, will provide means to limit the propagation of gender stereotypes by nmt systems when translating into languages with grammatical gender. the source code to reproduce our results for the publicly available data sets is published on github foot_5 .