Gender Bias Evaluation in Machine Translation for A mharic, T igrigna, and Afaan Oromoo (2024)

Abstract: While Machine Translation (MT) research has progressed over the years, translation systems still suffer from biases, including gender bias. While an active line of research studies the existence and mitigation strategies of gender bias in machine translation systems, there is limited research exploring this phenomenon for low-resource languages. The limited availability of linguistic and computational resources confounded with the lack of benchmark datasets makes studying bias for low-resourced languages that much more difficult. In this paper, we construct benchmark datasets to evaluate gender bias in machine translation for three low-resource languages: Afaan Oromoo (Orm), Amharic (Amh), and Tigrinya (Tir). Building on prior work, we collected 2400 gender-balanced sentences parallelly translated into the three languages. From human evaluations of the dataset we collected, we found that about 93% of Afaan Oromoo, 80% of Tigrinya, and 72% of Amharic sentences exhibited gender bias. In addition to providing benchmarks for improving gender bias mitigation research in the three languages, we hope the careful documentation of our work will help other low-resourced language researchers extend our approach to their languages.

## introduction machine translation (mt) systems play a pivotal role in breaking down language barriers and facilitating cross-cultural communication. gender bias poses a significant challenge, particularly in languages with limited linguistic resources. the imbalance within datasets used for mt training often results in genderrelated disparities. in low-resource languages like amharic, tigrinya, and afaan oromoo, and in morphologically rich languages like arabic @xcite professional names such as doctor, pilot, professor, etc., are mostly translated using the masculine gender. machine translation services often default to masculine forms for professions like "doctor" and "nurse," for feminine forms potentially reflecting and reinforcing gender stereotypes. figure 1 and figure 2 demonstrate this for the amharic language foot_0 . these types of bias can lead to misunderstandings and reinforce gender roles, influencing how people perceive different professions based on gender. understanding and addressing gender bias in mt systems is vital for ensuring equitable and accurate communication across diverse linguistic communities. addressing the issue of gender bias in mt systems requires adequate datasets for evaluation; a challenging task in the context of low-resource languages. this work contributes to building equitable mt systems for lowresource languages by constructing a goldtest dataset for three languages: amharic, figure 1 : translating the sentence "the doctor is coming" google translate translates the word "doctor" into masculine gender for the amharic language. the word "doctor," translated in amharic as "ዶክተር" (dokter), is gender-neutral. however, when translating "the doctor is coming," google translate translates the sentence to "ዶክተሩ እየመጣ ነው።" (dokteru eyemet'a new). here the phrase "the doctor" becomes "ዶክተሩ" (dokteru); the prefix "u" indicates masculine gender in the amharic language. in addition, the word "coming" translates into "እየመጣ" (eyemet'a); which also indicates masculine gender. figure 2 : in the sentence "the nurse is coming", the word "nurse," translated in amharic as "ነርስ" (ners), is gender-neutral. however, when translating "the nurse is coming," google translate translates the sentence to "ነርሷ እየመጣች ነው።" (nerswa eyemet'ach new). here the phrase "the nurse" becomes "ነርሷ" (ners-wa); the prefix "wa" indicates feminine gender in the amharic language. in addition, the word "coming" translates into "እየመጣች" (eyemet'ach); which also indicates feminine gender. ## related work investigating bias in mt systems is an active body of work in the nlp community. we use the taxonomy from @xcite and focus on representational harms due to stereotyping: sustaining stereotypical gender connotations for occupations during translation, thereby limiting the variety of occupations a specific gender may or may not engage in foot_1 . previous works in this space have relied on (1) curating benchmark datasets (e.g. @xcite ), (2) human evaluation schemes (e.g. @xcite ), and (3) automatic evaluation schemes(e.g. @xcite ). in curating benchmark datasets, @xcite prepared a gender-balanced dataset for evaluating gender bias in translation systems pertaining to occupation. since different languages represent gender in various ways @xcite , evaluation and mitigation strategies might also have to account for such variation. for instance, @xcite prepared test sets with gender natural pronouns used in the korean language for investigating bias in korean-english translation pairs. in evaluating gender bias in mt, several works rely on automatic metrics. @xcite found that google translate defaults to the masculine pronoun when translating job descriptions, particularly in relation to science, technology, engineering, and mathematics (stem) professions. @xcite introduces a new evaluation index, the translation gender bias index (tgbi), for measuring gender neutrality and evaluating korean-english translation pairs. @xcite introduce an evaluation protocol that relies on co-reference resolution datasets and morphological analysis to automatically evaluate gender bias across eight target languages that use grammatical gender. @xcite used the word embeddings fairness evaluation framework (wefe) to measure gender bias in mt systems built for luganda-english translation. while automated measures allow us to capture a broader understanding of the phenomenon, they may limit the detail and depth of our analysis. the study by @xcite uses automatic and human evaluations in tandem, exploiting both the versatility of automated evaluation and the nuance and detail captured by human evaluation. as the work by @xcite argues, it is important first to articulate how bias in such systems can be harmful. relying on the taxonomy of harms from prior work @xcite , we posit that understanding gender bias exhibited by mt systems would allow us to ( 1 ) uncover the representational harms the systems exhibit thereby understanding what power structures they uphold and (2) mitigate allocational harms that might result from deploying such systems in downstream applications (e.g. employment and job search). one challenge in studying bias in machinetranslated text is the diverse socio-cultural aspects that shape how gender is articulated among different groups and how stereotypes propagate in this diverse context. @xcite have shed light on the difficulty of studying and mitigating bias across multicultural, multilingual groups. such contexts require community-rooted efforts that thoroughly investigate how the culture and language are structured. in this work, we curate benchmark datasets for three low-resource languages through collaborations among native speakers. based on previous works, @xcite , we conduct an automatic evaluation of the translation quality overall and human evaluations of gender bias in popular machine translation systems to understand the current landscape of translation systems for these languages. ## background: linguistic gender representation amharic, a semitic language, uses grammatical gender. most nouns and pronouns have distinct masculine and feminine forms. genderspecific pronouns are used (e.g., እሱ (ǝssu) for "he" and እሷ (ǝsswa) for "she"), and job titles can also have gendered forms. like amharic, tigrinya, another semitic language, has grammatical gender. gender distinctions are marked in nouns and pronouns. there are specific pronouns for different genders (e.g., ንሱ (nǝssu) for "he" and ንሳ (nǝssa) for "she"), and job titles may vary depending on gender. afaan oromoo, a cushitic language, does not have grammatical gender in the same way as the other two languages. gender-neutral pronouns are often used, but context can some-times specify gender. gender is less likely to be marked in job titles compared to the other two languages of study. to illustrate more about the issues in translating a sentence and a professional word, we can see the following example of gender bias in english to amharic translation. the english to amharic google translate (accessed january 20, 2024) output of the sentence "the nurse helped the doctor" is "ነርሷ ሐኪሙን ረድታለች።" (nerswa hakimun redtalech). here, "ነርሷ" (nerswa) 'the nurse' is female, and "ሐኪሙን " (hakimun) "the doctor" is male. the word "helped", while it has a translation issue foot_2 , is translated to "ረድታለች" (redtalech), which is indicative of a feminine subject. in amharic, the source sentence "the nurse helped the doctor" can be translated in eight different ways as follows: here "ነርሷ" (neriswa) 'the nurse' is female, "ሐኪሙን" (hakimun) 'the doctor' is male, and "ረድታዋለች" (reditawalechi) 'she helped him'. here "ነርሷ" (neriswa) 'the nurse' is female, "ሐኪሟን" (ḥākīmwan) 'the doctor' is female, and "ረድታታለች" (reditatalechi) 'she helped her'. here "ነርሱ" (nersu) 'the nurse' is male, "ሐኪሟን" (hakimun) 'the doctor' is female, and "ረድቷታል" (reditwatale) 'he helped her'. 7. "ነርሱ ሐኪሟን ረድቷቸዋል።" (nersu ḥākīmwan reditwachewal). here "ነርሱ" (nersu) 'the nurse' is male, "ሐኪሟን" (ḥākīmwan) 'the doctor' is female, and "ረድቷቸዋል" (reditwachewal) 'he helped her' (for respect or plural). 8. "ነርሱ ሐኪሙን ረድቷቸዋል።" (nersu ḥākīmun reditwachewal). here "ነርሱ" (nersu) 'the nurse' is male, "ሐኪሙን" (ḥākīmun) 'the doctor' is female, and "ረድቷቸዋል" (reditwachewal) 'he helped him' (for respect or plural). this range of translations reflects the potential for gender bias in translation when assumptions are made about the gender of individuals based on their professional names. ## result and analysis figure 5 provides a clear comparison of responses across three language categories, allowing for insights into the distribution of responses within each language. it presents the gender bias across various language groups, delineating respondents' perceptions regarding the presence or absence of gender bias within each language category. the data in likewise, in the tigrinya group, the majority (80.96%) indicated observing gender bias, while 19.04% expressed no bias. these findings reveal distinct patterns regarding whether speakers observe gender bias across language groups, suggesting potential implications for addressing and understanding 1 outlines translation issues across languages, categorized into "translating the sentence issue" and "professional word translation issue." amharic records the highest instances of sentence translation issues at 1429, followed by tigrinya with 936, and afaan oromoo with 918. regarding professional word translation, afaan oromoo leads with 612 instances, trailed by tigrinya at 475, and amharic at 258. tigrinya exhibits the fewest reported issues overall, with 619 sentences indicating no translation issues, compared to 510 for amharic and 421 for afaan oromoo. conversely, amharic shows the highest incidence of respondents facing both types of issues at 203, followed by afaan oromoo at 449, and tigrinya at 370. this data underscores the diverse challenges faced in translation across languages and provides valuable insights for enhancing translation quality and addressing language-specific obstacles. the result shows that the google mt system outperformed the nllb model when using english as the source language in both evaluation metrics. this shows that translating english sentences into the target ethiopian language is challenging for the model. on the other hand, the google mt system showed better results compared to the nllb model when translating english sentences into target ethiopian languages. we observed better performance results when using english as the target language than when using it as the source language in the nllb model. from this, we can see that for low-resource languages, publicly available mt models like nllb are strug- ## conclusion and future work in this paper, we curated a benchmark dataset for evaluating gender bias in machine translation systems in three low-resource languages. with this test dataset, we conducted a humanlevel gender bias evaluation of google translate and nllb mt models for the given language pairs. the evaluation result shows that 92.96% of eng-orm, 80.96% of eng-tir, and 72.50% of eng-amh language pairs translations have a gender bias. in addition, we used the automatic evaluation to measure the translation quality of the currently available translation tools that support amharic, tigrinya, and afaan oromoo languages. our findings highlight the need for further research and development efforts to mitigate gender bias and promote gender-inclusive language translation. we observed that this work can be scaled up and used as a benchmark for other low-resource languages. in future work, we will use automatic gender bias evaluation metrics in addition to human evaluation. in addition, we will prepare a gender-balanced dataset for the given language, and we will fine-tune the currently available mt tools.