GFST : G ender-Filtered Self-Training for More Accurate Gender in Translation (2021)

Abstract: Targeted evaluations have found that machine translation systems often output incorrect gender in translations, even when the gender is clear from context. Furthermore, these incorrectly gendered translations have the potential to reflect or amplify social biases. We propose gender-filtered self-training (GFST) to improve gender translation accuracy on unambiguously gendered inputs. Our GFST approach uses a source monolingual corpus and an initial model to generate gender-specific pseudo-parallel corpora which are then filtered and added to the training data. We evaluate GFST on translation from English into five languages, finding that it improves gender accuracy without damaging generic quality. We also show the viability of GFST on several experimental settings, including re-training from scratch, fine-tuning, controlling the gender balance of the data, forward translation, and back-translation.

## introduction recent work has drawn attention to the harms that machine learning algorithms can cause by reflecting or even amplifying data biases against protected groups @xcite . for the most part, machine translation (mt) studies on bias have focused on gender bias in neural machine translation (nmt) and have identified a series of representational harms and stereotyping. 2 for example, on input sentences that are underspecified in terms of gender, mt models often default to masculine or gender-stereotypical outputs @xcite , which can exclude female and non-binary people (e.g., the sentence i am a doctor spoken by a woman may be translated incorrectly as i am a (male) doctor). even on unambiguously gendered inputs, nmt models can exhibit poorer performance, in terms of overall quality or gender translation accuracy, on content with non-masculine referents @xcite . in this paper, we take on the task of improving gender translation accuracy, focusing on unambiguous inputs where there is only one correct translation with respect to gender. this task is especially difficult when translating from languages with very limited grammatical gender (such as english) into languages with extensive gender markings (such as german). known sources of gender bias in mt include sample bias (a.k.a. selection bias), which occurs when the input (source) distribution differs from that of the target application; label bias, which in mt occurs when gender-neutral sentences are translated predominantly into a specific gender or when the gender is translated incorrectly in the training data; and over-amplification, which is a property of the machine learning model @xcite . in this paper we focus on sample bias, starting from the observation that mt training data is often en-de en-fr en-he en-it en-ru fem 0.7% 0.9% 1.9% 1.9% 0.5% msc 4.8% 3.1% 4.9% 4.9% 2.4% mix 94.5% 96.0% 93.2% 93.2% 97.1% this paper proposes a data augmentationbased method to address sample bias using only source-language monolingual data. our approach, dubbed gender-filtered self-training (gfst), consists of self-training the nmt model using genderbalanced monolingual data that is filtered to reduce error propagation. our framework is simple, generic, and easily scalable to any target language for which a morphological tagger is available. our main contributions are: in this paper, we propose gender-filtered selftraining (gfst) for improving gender translation 3 this work helps to mitigate representational harms caused by low gender translation accuracy in mt systems. since male and female genders have been the focus of most targeted mt gender bias evaluations, we focus on these two genders and as such do not address representational harms against nonbinary genders. see our impact statement in section 9 for more discussion. gfst assumes access to a parallel corpus d par and a monolingual source corpus d src . we first train an initial model θ ini on d par . due to the skewed gender representation of the training data (see therefore, we use θ ini to generate translations for gender-specific sentences extracted from d src . this forward-translated data is then filtered to ensure that the translations accurately reflect the gender of the source, balanced by gender, and used as additional training data. note that filtering is only done on the additional pseudo-parallel data; the original parallel data is used in its entirety. the full process is illustrated in algorithm 1, and we describe each step in detail below. algorithm 1 gfst for nmt. require: parallel and src mono data d par , d src 1: train θ ini on d par 2: for gen in {fem, msc} 3: d gen src ← filtersrc(d src , gen) 4: d gen trg ← translate d gen src using θ ini 5: d gen par ← filtertrg(d gen src , d gen trg , gen) 6: train θ fin on d par + d fem par + d msc par 7: return θ fin ## experiments with source language as english (en), we experiment on five target languages from four families, all of which have grammatical gender: french (fr), italian (it), russian (ru), hebrew (he), and german (de). our experiments include low-, mediumand high-resource settings. ## analysis 6.1 retraining vs. fine-tuning the main experiments (section 5) used the data generated by the gfst method to train the final models from scratch. in this section, we further explore the utility of gfst by fine-tuning the existing models that were used for forward translation, instead of retraining them. we fine-tune these models using the feminine and masculine samples, and additionally mix in an equal number of sentences from the original training corpus to avoid catastrophic forgetting (following the mixed fine-tuning approach of @xcite . bleu scores for baselines, retrained gfst models, and fine-tuned gfst models are shown in ## conclusion this paper addresses gender translation accuracy for unambiguously gendered inputs. the proposed gender-filtered self-training approach creates additional gender-specific training data by filtering source monolingual data by gender, translating the data, and filtering the translations to remove gender errors. using this additional data, the models achieve large gains in gender accuracy without damaging overall translation quality. in the future, we plan to extend gfst to other genders and language pairs. this will not be trivial: the self-training aspect of gfst assumes that the initial model is good enough at gender translation, which may not be the case for other genders and languages. in particular, the use of morphological analysis for filtertrg might limit gfst's applicability to other genders or very low-resource target languages. thus, we will explore alternative approaches to self-training (e.g. synthetic data generation) and filtering (e.g. using round-trip translation @xcite ).