Gender Bias Mitigation for NMT Involving Genderless Languages (2022)

Abstract: It has been found that NMT systems have a strong preference towards social defaults and biases when translating certain occupations, which due to their widespread use, can unintentionally contribute to amplifying and perpetuating these patterns. In that sense, this work focuses on sentence-level gender agreement between gendered entities and occupations when translating from genderless languages to languages with grammatical gender. Specifically, we address the Basque to Spanish translation direction for which bias mitigation has not been addressed. Gender information in Basque is explicit in neither the grammar nor the morphology. It is only present in a limited number of gender specific common nouns and person proper names. We propose a template-based fine-tuning strategy with explicit gender tags to provide a stronger gender signal for the proper inflection of occupations. This strategy is compared against systems fine-tuned on real data extracted from Wikipedia biographies. We provide a detailed gender bias assessment analysis and perform a template ablation study to determine the optimal set of templates. We report a substantial gender bias mitigation (up to 50% on gender bias scores) while keeping the original translation quality.

## introduction as the neural machine translation (nmt) field becomes more mature, there is a growing concern about the gender fairness of these systems @xcite @xcite . these data-driven approaches are trained on large real-world textual corpora which often exhibit implicit social gender stereotypes and biases. for example, @xcite noted that systems associate certain neutral occupations with males, such as doctor or programmer, and others with females, such as nurse or housekeeper. as a consequence, although not being required by the task, systems tend to inherit and amplify these social biases. several different solutions have been proposed to solve, or at least reduce, gender bias during the translation process: providing alternative masculine and feminine translations for some neutral words @xcite ; adding explicit gender information during training @xcite @xcite ; removing bias from word embeddings @xcite ; or fine-tuning on a small gender-balanced data set (costa-jussà and de jorge, 2020; @xcite . there have also been some efforts to construct some challenge sets to systematically assess gender bias @xcite . most of the previous work has focused on english as the source language which is then translated to languages with grammatical gender such as spanish, french, german, etc. english is a notional gender language which encodes gender in a pronominal system (he/she, his/her...) @xcite . consolidated evaluation benchmarks such as winomt @xcite or must-she @xcite are specially designed for english. however, for genderless languages such as basque, existing previous work and benchmarks do not fully satisfy the requirements. for example, winomt uses pronominal references as a disambiguation signal for the correct inflection of occupations, which do not exist in basque. gender information in basque is explicit in neither the grammar nor the morphology. this fact implies that gender can only be determined when nouns correspond unequivocally to a female or a male, that is, person proper names or a limited number of gender-specific common nouns (e.g., emakumea/gizona, aita/ama... 1 ), hereinafter referred to as gendered entities. therefore, existing approaches and evaluation benchmarks need to be figure 1: an illustrative example for the task of sentence-level agreement between gendered entities and occupations when translating from genderless languages to gendered languages. english translation: mikel wants to be a nurse. in this work we address the specific task of sentence-level gender agreement between gendered entities and occupations when translating from genderless languages to gendered languages (see example in figure 1 ). we focus on the basque to spanish translation direction, a translation direction that presents the peculiarities described above and that has not been studied in the literature. the main contributions of the paper are the following: ## related work a wide variety of publications warn about the lack of fairness some of the commercial mt systems have and how they might contribute to amplify and perpetuate social gender stereotypes due to their widespread use @xcite @xcite . in the case of the basque-spanish language pair, @xcite found a preference for the stereotyped translation of occupations according to their historically assigned role. ideally, system bias could be mitigated by removing all the bias present in the training data. for example, by augmenting data samples with their corresponding counterfactual forms @xcite . nevertheless, this task still poses some challenges for grammatical gender languages as it involves preserving the morphosyntactic agreement of the whole sentence by gender swapping pronouns, adjectives, verbs, entities, etc., @xcite . as a result, several alternative methods have been proposed to alleviate gender bias from mt systems. @xcite add a special gender token to source sentences in order to improve morphological agreement between the uttered sentence and the gender of the speaker. stafanovičs et al. (2020) 2020 ) provide the system with discourse context by adding the previous sentence, and in the same direction, @xcite propose a method to guide a black-box model by appending some contextual gender unambiguous hints to source sentences, such as "... she told them". other more complex approaches have targeted gender bias effects by directly equalizing genders in word embeddings @xcite . another promising line of research addresses gender bias as a domain adaptation problem by finetuning a pre-trained biased system with a genderbalanced data set. costa-jussà and de jorge (2020) automatically collect gender-balanced parallel data from wikipedia biographies by selecting an equal amount of examples for each gender. @xcite generate gender filtered parallel data by forward-translating a monolingual corpus. saunders and byrne (2020) generate a small, trivial, gender-balanced set of synthetic examples by inflecting a single handcrafted template with an equal number of masculine and feminine entities. in @xcite they further improve the method by adding explicit word-level gender tags. our proposed method uses a controlled genderbalanced set of examples (section 3) to fine-tune a pre-trained model (section 4.1). we further provide the system with explicit gender tags for proper gender inflections (section 4.2). we assume that providing a stronger gender signal is better than letting the system infer the proper gender of each gendered entity. in order to annotate source words, @xcite propose a complex method involving morphological tagging and automatic alignment, and @xcite rely on the proper coreference resolution. in contrast, we directly annotate gendered entities and leave gender agreement to the system instead of directly providing gender inflection information for all the affected words. this assumption simplifies the annotation effort as only gendered entities lists are necessary to annotate the data. ## gender-balanced corpora existing previous work on gender bias has focused on high resource translation directions (english to spanish, french, german, etc.). while consolidated benchmarks and data exist for these languages, they strongly rely on gendered pronouns which makes them difficult to adapt for basque. thus, we analyzed two different strategies to build gender-balanced corpora for the basque to spanish translation direction: a syntactically diverse set of handcrafted templates (section 3.1), and real data extracted from wikipedia biographies (section 3.2). ## experimentation all the systems use the default configuration for the transformer architecture @xcite as implemented in the pytorch version of the open-nmt toolkit @xcite . we apply bpe tokenization @xcite trained on 32,000 operations on the joint training data. sentences larger than 100 tokens are discarded from the training set. the baseline systems were trained on the basque-spanish portion (1.77m examples) of the paracrawl (v8) data @xcite . the gender-balanced systems are trained by fine-tuning the baseline system on the gender-balanced data sets described in section 3. as in costa-jussà and de jorge (2020), to avoid catastrophic forgetting, where systems tend to forget about previous knowledge, we follow a mixed fine-tuning strategy @xcite . a weighted combination (10:1 ratio 8 ) of general domain data from paracrawl and task specific data, such as templ_train or wiki_train, is used during training and validation steps. for validation purposes, we concatenate 5,000 general domain examples and 1,000 task specific examples randomly extracted from the training data. the baseline and the fine-tuned systems have been trained until convergence on the perplexity results on the validation set, stopping the training process if there was no improvement for 5 consecutive checkpoints. validation is performed every 10,000 steps in the case of the baseline system whereas fine-tuning validation is performed every 1,000 steps. we evaluate our systems using bleu and chrf++ scores from the sacrebleu tool @xcite . additionally, we also provide comet @xcite ) scores foot_7 , a metric which focuses on the semantic similarity by leveraging the recent breakthroughs in neural language modeling. these scores are computed on the test sets extracted from three publicly available corpora: eitb (etchegoyhen and gete, 2020) a news domain data set, ehuhac @xcite a collection of classic books, and ted (reimers and gurevych, 2020) comprising ted talks transcriptions. from each set, we randomly extracted 5,000 examples. although bleu, chrf++ and comet metrics measure the overall translation quality of the systems, task specific metrics are required to evaluate gender bias with more precision. to that end, we measure the accuracy of the correctly translated and gender inflected occupations and we propose a new metric called swap. swap is defined as the percentage of the occupations which are inflected with the opposite gender. thus, errors are divided into unrecoverable errors where occupations are translated in a different way (errors) and gender swapped occupations (swap). a higher swap score means higher bias towards the opposite gender. these scores are computed on the task specific test sets mentioned in section 3.