Gender bias Evaluation in L uganda- E nglish Machine Translation (2022)

Abstract: We have seen significant growth in the area of building Natural Language Processing (NLP) tools for African languages. However, the evaluation of gender bias in the machine translation systems for African languages is not yet thoroughly investigated. This is due to the unavailability of explicit text data available for addressing the issue of gender bias in machine translation. In this paper, we use transfer learning techniques based on a pre-trained Marian MT model for building machine translation models for English-Luganda and Luganda-English. Our work attempts to evaluate and quantify the gender bias within a Luganda-English machine translation system using Word Embeddings Fairness Evaluation Framework (WEFE). Luganda is one of the languages with gender-neutral pronouns in the world, therefore we use a small set of trusted gendered examples as the test set to evaluate gender bias by biasing word embeddings. This approach allows us to focus on Luganda-Engish translations with gender-specific pronouns, and the results of the gender bias evaluation are confirmed by human evaluation. To compare and contrast the results of the word embeddings evaluation metric, we used a modified version of the existing Translation Gender Bias Index (TGBI) based on the grammatical consideration for Luganda.

## introduction uganda is a highly multilingual country with over 43 known indigenous languages and dialects @xcite . however, many languages have no existing resources for building natural language processing (nlp) datasets and tools. one of the major languages spoken in uganda is luganda, primarily spoken in the south eastern buganda region, mainly along the shores of lake victoria and up north towards the lake kyoga shores @xcite . luganda is spoken by more than six million people, principally in central uganda, including kampala, the capital of uganda. topologically, it is a highly agglutinating, tonal language with subject-verb-object, word order, and nominative-accusative morphosyntactic alignment @xcite . like many languages in sub-saharan africa, luganda has limited text and speech data resources, making it a low-resourced language. however, due to the increase in the availability of computational resources and datasets, there has been high advancement in nlp research. natural language processing applications approach tasks ranging from low-level processing, such as assigning parts of speech to words, to high-level tasks, such as question answering, and translating speech and text from one language to another. in this paper, we focus on building and evaluating machine translation (mt) models for luganda as an application of natural language processing. classically, rule-based systems were used for mt, these were replaced with statistical methods @xcite in the 1990s. more recently, deep neural network models have achieved state-of-art results in the field of neural machine translation (nmt) @xcite . with neural machine translation and language processing tools becoming more prevalent, there has been a high interest in understanding and mitigating bias in nlp systems. this is because nlp systems are considered susceptible to social bias @xcite . the investigation of bias is not only a scientific and technical endeavour but also an ethical one, given the growing role of nlp applications @xcite . bias in mt is when an mt model systematically and unfairly discriminates against certain individuals or groups in favour of others @xcite . since mt systems are used daily by millions of individuals, they could impact a wide array of people in different ways @xcite . in the real world, the highest form of bias in machine translation systems is gender bias, which manifests itself when training data has more features and examples of a given gender stereotype compared to others. machine translation (mt) tools trained on such data inherit the existing biases in the data. different languages deal with gender in different ways; for instance, some languages have gendered pronouns like he/she/him/her in english, @xcite whereas others, such as luganda, finnish, hungarian, turkish, etc. have neutral pronouns @xcite . unlike machines, a human translator can understand what the correct translation should be depending on the context. however, this can be complex with machine translation (mt) tools, except the models are contextualized to a specific domain. the gender bias problem occurs when the mt engine has to pick one pronoun over another, as dictated by the noun. gender bias in machine translation for gender-neutral languages is one of the most complex forms of bias. a case in point is when translating from luganda-english; all gender-neutral pronouns are translated into gender-specific nouns. for example, consider the luganda sentence below: musawo mu ddwaliro ly'e mulago. is translated to a gender-specific sentence: he is a doctor at mulago hospital, we show that the word he is neutral in luganda hence being nonexistent. the word musawo is translated to doctor, the word mu is translated into two english words is and a respectively, while the words ddwaliro, and ly'e are translated into hospital and at respectively as shown in the ## related work in this section, we review related work in machine translation (mt) of low-resourced languages, the models used, and the evaluation of gender bias in machine translation (mt). neural machine translation (nmt) has seen a tremendous growth spurt in less than ten years. while considered the most widely used solution for machine translation, its performance on lowresource language pairs remains suboptimal compared to the high-resource counterparts, for example, english, german, and spanish, among others, due to the unavailability of large parallel corpora. therefore, the implementation of nmt techniques for low-resource language pairs has been receiving the spotlight in the recent nmt research arena, thus leading to a substantial amount of research @xcite . prior work in machine translation (mt) with a focus on low-resource languages has been building language corpora and baseline models. the lack of training data motivated research to compare zero-shot learning, transfer learning, and multilingual learning of three bantu languages (shona, isixhosa, and isizulu) and english @xcite . in the study on neural machine translation (nmt) for african languages, the authors to @xcite address the problems of the lack of datasets required for machine translation and existing research to reproduce the work on african languages. @xcite presents the menyo20k yoruba-english language with standardized train-test splits for model benchmarking. researchers are leveraging several sources of text data for creating datasets focused on news headlines and text sources in their local context @xcite . the authors in @xcite propose a participatory approach to building parallel corpora and mt models to deal with the lack of language resources. based on an ongoing lacuna-funded project, an effort has been made to build parallel text corpora for five ugandan languages, i.e., luganda, runyokore-rukiga, acholi, and lumasaaba @xcite . there has been advancement in building machine translation (mt) models and datasets by leveraging pre-trained and multilingual models. research that involved building datasets for african languages and researchers adapted several multilingual pre-trained baseline models (ifeoluwa @xcite . work has been done in which multilingual parallel corpora were created for five (5) ugandan languages and carried out on neural machine translation (nmt) models to build baseline multilingual models @xcite . while translation technologies bring undeniable advantages in many contexts, it is also evident that they come with inherent risks, such as reproducing and even amplifying real-world asymmetries by codifying and entrenching various kinds of biases. one of the biases is gender bias, which affects automatic translation. this is also seen when systems are required to overly express gender in the target languages while translating from languages that do not convey such information @xcite . in the paper by @xcite , the authors present the research carried out to understand, assess and mitigate gender bias in automatic translation. the study discusses how the socio-cultural notions of gender interact with language(s) and translation and frames, which factors can contribute to the emergence of gender bias in automatic translation systems. they present the resources created to assess the biased behaviour of mt systems and the mitigation strategies developed to reduce feminine under-representation in their outputs. the authors @xcite treat gender as a domain for machine translation, training from scratch by augmenting europarl data with a tag indicating the speaker's gender. this does not inherently remove gender bias from the system but allows control over the translation hypothesis of gender. work in @xcite evaluates and quantifies the gender bias within a hindi-english machine translation system, and they implement a modified version of the existing one. translation gender bias index (tgbi) metric is based on grammatical considerations for hindi. they compare the results of word embeddings fairness evaluation (wefe) framework metrics with the pre-trained word embeddings and the ones learned by their machine translation model. to our knowledge, there is not a lot of work done on evaluating gender bias in translation languages with gender-neutral languages. in a research study by @xcite , gender bias is measured in the translation of gender-neutral pronouns using translation bender bias index (tgbi). in the tgbi metric, the authors quantify the associations of "he", "she" and other related gendered words in the translated text. in this paper, we used the word embeddings fairness evaluation (wefe) framework metrics to evaluate gender bias on word embeddings learned by our translation system. this is because word embeddings exhibit stereotypical bias towards gender, race, religion, ethnicity etc @xcite . we also used a modified version of the tgbi metric on luganda, a low-resourced language with gender-neutral pronouns. ## corpus creation in this section, we describe the process taken to create the luganda to english parallel corpus used for training and evaluating gender bias in the machine translation model.