Evaluating the Impact of Stereotypes and Language Combinations on Gender Bias Occurrence in NMT Generic Systems (2023)

Abstract: Machine translation, and more specifically neural machine translation (NMT), have been proven to be subject to gender bias in recent years. Many studies have focused on evaluating and reducing this phenomenon, mainly through the analysis of occupational nouns’ translation for the same type of language combinations. In this paper, we reproduce a similar test set than in previous studies to investigate the influence of stereotypes and language combinations’ nature (formed with English, French and Italian) on gender bias occurrence in NMT. Similarly to previous studies, we confirm stereotypes as a major source of gender bias, especially in female contexts, while observing bias even in language combinations traditionally less examined.

## introduction recently, gender bias in natural language processing (nlp), and more specifically in machine translation (mt), have been a raising concern in the research field @xcite as such phenomenon can lead to allocation and representational harms @xcite . yet, bias in machine learning (ml) is not a new phenomenon. in 1996, friedman et nissenbaum described it as "computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others" @xcite . @xcite also identified several sources responsible for bias occurrence:  preexisting bias. bias already existing in the training data on which the system is built and trained. as mentioned by @xcite , these different factors influencing bias in machine learning should not be seen as autonomous elements. the different factors are tightly interlinked and may even reinforce one another. in mt, one additional potential source of gender bias is the difference between languages. gender is not expressed in the same way in every language. based on @xcite and @xcite , we identify three types of languages: over the years, several test sets were developed to evaluate gender bias nlp systems, mainly based on the test suites models @xcite , updated in works such as @xcite . these specific test sets are identified as gender bias evaluations testsets (gbets) by @xcite . in the vast majority, gbets in machine translation evaluate gender bias through the study of occupational nouns (and adjectives) and are based on a binary vision of gender. however, despite numerous similarities in gbets, they can be divided into different categories according to their methodology @xcite : they focus on studying sentences, phrases or words in which no gender is defined in the source language translated into a language in which gender has to be marked. translation gender bias index @xcite is an example. despite using different approaches, three conclusions have emerged in gender bias literature: gender bias do occur in translations produced by mt, typically neural machine translation (nmt); they seem to be highly motivated by gender stereotypes; and mt systems tend to have a male default @xcite -they tend to favor masculine forms at the expense of feminine forms (e.g. @xcite @xcite . beyond these common results, another aspect is important to notice in gender bias literature: the vast majority of studies focus on language combinations in which the translation is made from a language with no or little gender markers into a language with more gender markers, as if this translation difficulty was required to analyse gender bias in machine translation. in most cases, the language combinations studied were either from a genderless language into english (notional gender language) or from english into a grammatical gender language. however, @xcite observed gender bias in translations from french into english. similarly, @xcite were able to study covert gender bias in translations from english into turkish (genderless language), as well as @xcite observed gender bias from and into french and italian, two grammatical gender languages very close in their gender marking. these results are proof that language combinations different from the ones usually studied are worth being further examined. following these observations, this study aims at contributing to gender bias understanding, and focuses more specifically on investigating the influence of stereotypes and language combinations on bias occurrence in generic nmt systems. our contribution includes a test suite with 40 sentences formed with occupational nouns and unambiguous gender markers, studied in six different language combinations, and which translations were analysed through human evaluation. in section 2, we will introduce our experimental framework, followed by our results in section 3. finally, section 4 will be dedicated to our conclusions and propositions for further work. ## experimental framework in this section, we will describe how our test set was created (section 2.1), how the translation was conducted (section 2.2), and how the translated data was evaluated (section 2.3). ## results in this section, we will explore our results, first analysing the influence of stereotypes on gender bias occurrence (section 3.1), then the influence of language combinations on the phenomenon (section 3.2). overall, our results have shown less biased translations than expected: the general error rate is only 13.6% (see ## conclusions and further work this experiment has confirmed, as seen in previous studies, that stereotypes are undeniably the main source of gender bias in generic nmt. it has also shown that gender bias tends to occur more frequently in female contexts, which echoes the phenomenon of male default @xcite discussed in previous studies (e.g. @xcite @xcite . as for language combinations, the experiment has shown that gender bias is not a phenomenon specific to combinations from a language less marked in terms of grammatical gender into a language more marked, such as @xmath0. indeed, just as in @xcite , data has shown the presence of gender bias even in translations between french an italian, two languages with identical gender systems, despite their grammatical proximity and the unambiguous aspect of our benchmark. overall, no clear typology has been detected according to language combinations' nature, but rather noticeable results individually for the combinations at our ranking's extreme ends. moreover, except for stereotypes, we have not been able to clearly identify the source of specific system's behaviours regarding gender bias. yet, potential sources were still suggested and have to be taken into account, such as training corpora quality and quantity (especially in italian) or syntactic features (pronominal reference). these considerations exemplify the complexity of bias phenomenon and its multiple interlinked sources. also, some directions would be worth following for further research.