GATE X - E : A Challenge Set for Gender-Fair Translations from Weakly-Gendered Languages (2024)

Abstract: Neural Machine Translation (NMT) continues to improve in quality and adoption, yet the in advertent perpetuation of gender bias remains a significant concern. Despite numerous studies on gender bias in translations into English from weakly gendered-languages, there are no benchmarks for evaluating this phenomenon or for assessing mitigation strategies. To address this gap, we introduce GATE X-E, an extension to the GATE (Rarrick et al., 2023) corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. Each translation is accompanied by feminine, masculine, and neutral variants. The dataset, which contains between 1250 and 1850 instances for each of the four language pairs, features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. Additionally, we present a translation gender rewriting solution built with GPT-4 and use GATE X-E to evaluate it. We open source our contributions to encourage further research on gender debiasing.

## introduction despite dramatic improvement in general nmt quality and breadth of supported languages over recent years @xcite , gender bias in nmt output remains a significant problem @xcite . one such type of gender bias is spurious gender-markings in nmt output when none were present in the source. this occurs most frequently when translating from a weakly-gendered language into a more strongly gendered one. we explore this phenomenon in translations from turkish, persian, finnish, and hungarian into english. gender can be marked in english through gendered pronouns (he, she, etc.) and possessive determiners (his, her), or through a limited number of çocuklar için öğle yemeği hazırladı. she prepared lunch for the children. she prepared lunch for the children. (f) he prepared lunch for the children. in each of our selected set of source languages, all personal pronouns are gender-neutral, such as o in turkish meaning he/she/singular they. these languages do use some intrinsically gendered noun words, but not necessarily for all of the same concepts that english does. turkish differentiates mother (anne) from father (baba), but does not differentiate nephew from niece (both are yegen). this difference in gender on third-person singular pronouns leads to translation scenarios such as the one seen in figure 1 , where someone with no specified gender in the source is marked as female in the translation through the pronoun she. nmt models often make gender assignments according to stereotypes @xcite -in this case a model appears to associate child care with women. one remedy for this category of problems is to supplement the default feminine translation with masculine and genderneutral alternatives, so that all possible gender interpretations are covered. this can be accomplished by applying a gender rewriter to the original nmt output, as shown in the bottom portion of figure 1 . gate (gender-ambiguous translation examples, @xcite ) introduced an evaluation benchmark for gender rewrites for translations from english into french, spanish, and italian. in this work, we introduce gate x-e 2 , an extension to gate that focuses on translations into english from a set of more weakly-gendered languages. it consists of natural sentences with strong diversity of sentence lengths and domains, and challenges translation rewriters on a wide range of linguistic phenomena. gate x-e contains between 1250 and 1850 instances for each of our language pairs. we also present a translation-rewriting solution that utilizes @xcite to provide gendered and gender-neutral alternatives. it achieves high accuracy on the pronoun-only subset of gate x-e. finally, we also perform human evaluation and provide a detailed error analysis of the results. the remainder of this paper is organized as follows -in section 2, we discuss the corpus creation process and structure of gate x-e. in section 3, we discuss how various properties can affect the difficulty of translation rewriting problems. in section 4 we introduce a gpt-4-based translation-rewriting solution. in section 5 we discuss results of our experiments and perform detailed error analysis. finally, in section 6 we cover related work. ## gate x-e dataset we introduce gate x-e by describing the annotation process and labels used, as well as providing statistics on the collected data. ## translation gender rewriting translation gender rewriting is the process of taking a translated source-target pair and producing alternative translations with different gender markings. in a correctly rewritten translation, the gender markings should remain compatible with all gender information found in the source sentence @xcite . we consider this problem from the viewpoint of a user who wishes to see a set of three gendered-alternative translations with uniform output gender side-by-side: all-female, all-male and all-neutral. because the translations will be viewed as a set, the translations should only vary from one-another in specific words that mark gender. here we discuss the difference in difficulty between rewrite problems where gendered nouns are included and those where gender is only marked by pronouns. ## experiments we leverage gpt-4 to implement a translation gender-rewriting solution and evaluate it on gate x-e. ## results following @xcite , we focus on exact match accuracy to the reference. frequently only one or two words will be different between an original translation and a correct rewrite. in this context, metrics such as bleu @xcite and wer are not very effective at determining the significance of single extraneous or missed word modifications. therefore, credit is only given for a test tuple when the rewriter output exactly matches the reference output. ## related work understanding and assessing gender bias: it has been documented that mt systems often make mistakes and show gender biases when translating between languages with differing gender norms @xcite @xcite @xcite @xcite . @xcite show an 8-bleupoint gap between masculine and feminine references when translating from non-gender-marked english sentences into various strongly-gendered langauges. měchura (2022) proposes a taxonomy for describing situations where properties such as gender, number or formality are unknown in a source sentence but assumed in a translation, resulting in bias. ## conclusion we have presented gate x-e, a diverse dataset covering a wide range of scenarios relevant to translation gender-rewriting for english-target language pairs, covering gendered and gender-neutral rewrites. we have discussed intricacies of englishtarget translation rewriting, and explained what properties lead to easier or more difficult rewrite problems. we have explored the ability of gpt-4 to provide rewrites for these translations, showing that it can achieve very high accuracy on pronounonly rewriting problems, but performs less well when gendered nouns are introduced. in the future we hope to expand the set of source languages beyond the four weakly gendered languages used in this work, to include those with different gender nuances. as can be seen from gpt-4's inconsistent performance in translation genderrewriting, particularly when gendered nouns are introduced, additional methods need to be explored in order to achieve sufficiently accurate and unbiased translation alternative sets. we also hope that by making gate x-e accessible to the broader research community, we can encourage further research on gender debiasing in the machine translation space.