Reducing Gender Bias in NMT with FUDGE (2023)

Abstract: Gender bias appears in many neural machine translation (NMT) models and commercial translation software. Research has become more aware of this problem in recent years and there has been work on mitigating gender bias. However, the challenge of addressing gender bias in NMT persists. This work utilizes a controlled text generation method, Future Discriminators for Generation (FUDGE), to reduce the so-called Speaking As gender bias. This bias emerges when translating from English to a language that openly marks the gender of the speaker. We evaluate the model on MuST-SHE, a challenge set to specifically evaluate gender translation. The results demonstrate improvements in the translation accuracy of the feminine terms.

## introduction when we talk about gender bias in neural machine translation (nmt), the first issue that comes to mind is stereotyping, e.g. associating the profession doctor with the male pronoun and nurse with the female pronoun. while this example does illustrate a clear instance of gender bias, @xcite highlight that it is crucial to recognize that gender bias can manifest in various forms. it becomes essential to determine precisely what is considered harmful, the manner in which it is perceived as harmful, and the specific individuals or groups affected @xcite . current research on mitigating gender bias in mt often focuses on gender stereotypes figure 1 : this work focuses on the speaking as bias emerging when translating from english to a language that openly marks the gender of the speaker as here in italian for instance: sono sicuro/a ... which translates to "i'm certain ..." @xcite , translation errors due to speaker gender @xcite , or pronoun translation @xcite . furthermore, the proposed methods are often only evaluated on bleu @xcite . however, bleu evaluates on word level and is rather insensitive to specific linguistic phenomena that only affect a few words @xcite . in this paper, we apply a controlled text generation method, future discriminators for generation (fudge) @xcite , to mitigate the gender bias that arises when translating from english, a language that marks gender only on pronouns, to italian, a language that openly marks the gender of the speaker in specific contexts. fudge has demonstrated its capabilities on many controlled text generation tasks, e.g. poetry couplet completion, topic-controlled language generation, and machine translation formality change. we further explore fudge's performance on a gendercontrolled machine translation task. furthermore, instead of solely relying on bleu @xcite as an evaluation metric, we evaluate on must-she @xcite , a novel gender translation challenge set that was built on manually annotated test sets and is specifically designed to measure the translation accuracy of gendered expressions. fudge demonstrates improvements in several feminine gender terms' translation accuracy. 1 ## bias statement in this paper, we explore how to mitigate the mistranslation of feminine gender terms into masculine forms when translating from english to a language that openly marks the gender of the speaker. when an nmt system systematically assumes the gender of the speaker is male, this will cause representational harm, resulting in frequent translation errors for female speakers. we borrow the systematic classification proposed by @xcite , which classifies gender bias into three dimensions: speaking about (gender of the topic), speaking as (gender of the speaker), and speaking to (gender of the addressee). in this work, we focus on the speaking as bias, which usually appears in first-person sentence translations. due to the limitations of annotated data sets, we can only experiment on sentences by male and female speakers. more in-depth research on reducing the representational bias towards non-binary speakers will be possible. ## related work controlled text generation some research focuses on fine-tuning a pre-trained model for a desired attribute. @xcite propose a framework for neural natural language generation (nnlg) controlling different stylistic aspects of the generated text. the method results in a class-conditional language model (cclm), but it is difficult to separate the desired attribute from the generation model, i.e. the model is usually suitable for one task and needs retraining for another attribute of interest. @xcite mitigate this issue by proposing a conditional transformer language (ctrl) model that is conditioned on many factors including style, content, and taskspecific behavior. however, this is quite expensive. 1 code and documentation for the experiments are available on https://github.com/tianshuailu/debias_ fudge. @xcite suggest using discriminators to guide the decoding of lms. @xcite propose mucoco foot_0 where they formulate the decoding process as a continuous optimization problem that allows for multiple attributes. gender debiasing a common method to mitigate gender bias is to attach gender tags as proposed by @xcite . in this case, gender information is integrated into the nmt systems via a tag on the source side. this approach achieves improvements for multiple language pairs. given the original biased data set, @xcite propose to construct an additional training corpus where all male entities are swapped for female entities and vice-versa. the goal of the augmentation is to mitigate the bias by training the model on gender-balanced data sets. gender bias evaluation bemchmarks @xcite introduce a benchmark, winobias, to measure gender bias in coreference resolution with entities corresponding to people referred to by their occupation. another benchmark, wino-gender @xcite , is a winograd schema-style @xcite set of minimal pair sentences that differ only by pronoun gender. based on winobias and winogender, @xcite compose a coreference resolution english corpus that contains sentences in which the subjects are in non-stereotypical gender roles. it is a standard test set to evaluate gender stereotyping in mt. in contrast, must-she @xcite provides a fine-grained grammatical gender evaluation on word level and gender agreement level, which makes it more suitable to evaluate our model. ## method in a controlled text generation task, it is usually nontrivial to retrain the model g to condition it on the new attribute a. @xcite propose future discriminators for generation (fudge), a flexible and modular way of conditioning the generative model g on the desired attribute a that only requires access to the output probabilities of g. fudge achieves this by training a binary classifier that predicts at each time step t whether the attribute a will be satisfied in the complete sequence, based on the already generated tokens y 0 -y t . to see if gender tags improve fudge's performance, we have two underlying english-italian translation models, g and g t . we train both models on the same sentence pairs, with the exception that g t 's data set includes gender tags on the english source side. the method of adding gender tags is inspired by @xcite . the desired attributes are feminine and masculine, hence we train two classifiers b f and b m . each of them is combined with the two underlying translation models g and g t , resulting in four combinations, as illustrated in figure 2 . an advantage of fudge is the fact that it only needs access to the output logits of the generator model, meaning g and g t can be directly combined with b f and b m without additional fine-tuning or modification. this allows us to directly use g and g t as baselines. parlamint 2.1 is a multilingual set of 17 corpora containing parliamentary debates, including gender tags @xcite . in italian, the adjectives and participles are marked with the gender of the speaker in certain grammatical contexts. in the full data set, the utterances where the gender of the speaker is marked are relatively sparse. hence, we removed sentences that do not contain adjectives or participles for these experiments, since these cannot be marked for the gender of the speaker. the sizes of the original data set and the amount we used are shown in must-she v1.2 is a multilingual benchmark allowing for a fine-grained analysis of gender bias in machine translation and speech translation @xcite . must-she v1.2 contains 656 first-person sentences out of 1073, which makes it suitable for the evaluation of fudge. ## conclusion we explore controlled text generation in the context of gender bias by utilizing future discriminators for generations (fudge) @xcite in combination with a pre-trained model, mt5-small. our experiments show that baseline models generally work well on masculine forms since those are much more frequent in the training data