Test Suites Task: Evaluation of Gender Fairness in MT with M u ST - SHE and INES (2023)

Abstract: As part of the WMT-2023 “Test suites” shared task, in this paper we summarize the results of two test suites evaluations: MuST-SHEWMT23 and INES. By focusing on the en-de and de-en language pairs, we rely on these newly created test suites to investigate systems’ ability to translate feminine and masculine gender and produce gender-inclusive translations. Furthermore we discuss metrics associated with our test suites and validate them by means of human evaluations. Our results indicate that systems achieve reasonable and comparable performance in correctly translating both feminine and masculine gender forms for naturalistic gender phenomena. Instead, the generation of inclusive language forms in translation emerges as a challenging task for all the evaluated MT models, indicating room for future improvements and research on the topic. We make MuST-SHEWMT23 and INES freely available.

## introduction as machine translation (mt) has made strides in generic performance, there is an increasing recognition of the need to scrutinize finer, more nuanced aspects that defy assessment through traditional metrics computed on generic test sets. it is within this context that the wmt test suites shared task emerges, aiming to provide a dedicated evaluation framework to delve into specific dimensions of mt output with a laser focus. in particular, those representing well-known challenges within the current mt landscape. in light of the above, our contribution is dedicated to the critical themes of gender bias and inclusivity in translation @xcite . given the large-scale deployment of mt, such aspects are not only relevant from a technical perspective, where gender-related errors negatively impact the accuracy of automatic translation. rather, biased and non-inclusive systems can pose the concrete risk of under/misrepresenting gender minorities by over-producing masculine forms, while reinforcing binary gendered expectations and stereotypes @xcite . accordingly, in this paper we present the fbk participation in the test suites shared task by conducting evaluations on two newly-created test suites: the must-she w m t 23 and ines datasets, as well as their corresponding metrics and evaluations, are respectively discussed in section 2 and 3. in the evaluations presented in this paper, we obtained translations of our test suites by systems that are part of the standard general translation task of the eighth conference on machine translation (wmt-2023). in particular, we evaluated 11 systems for must-she w m t 23 en-de and 13 systems for ines de-en. ## must-she w m t 23 : en-de evaluation must-she w m t 23 is a test suite designed to evaluate the ability of mt systems to correctly translate gender. it is composed of 200 segments that require the translation of at least one english gender-neutral word into the corresponding masculine or feminine target word(s) in german. the test suite is created as an extension of must-she, a multilingual, natural benchmark built on ted talks data @xcite , which allows for a fine-grained analysis of gender bias in mt and st. the original must-she corpus comprises ∼3,000 (audio, transcript, translation) triplets annotated with qualitatively differentiated gender-related phenomena for thee language pairs: @xmath0. here, we introduce a newly created @xmath1, translation) of the must-she corpus. ## ines: de-en evaluation the inclusive evaluation suite (ines) is a test set designed to assess mt systems ability to produce gender-inclusive translations for the @xmath2. by design, each german source sentence in ines includes an expression that can be rendered by means of either an inclusive (in) or non-inclusive (n-in) expression in the english target language. overall, ines comprises 162 manually curated german sentences, which are annotated with their corresponding (in/n-in) english expressions. as such, it allows to evaluate to what extent mt systems favor the generation of non-inclusive solutions over alternative, valid inclusive translation in their output. ## related work the last few years have witnessed and increasing attention toward (binary) gender bias in nlp @xcite @xcite . concurrently, emerging research has highlighted the importance of reshaping gender in nlp technologies in a more inclusive manner @xcite , also through the representation of non-binary identities in language @xcite . foundational works in this area have included several applications, such as coreference resolution systems (cao and @xcite and fair rewriters @xcite . in mt, the research agenda has mainly focused on the improvement of masculine/feminine gender translation into grammatical gender languages @xcite . along this line, different strategies have been devised to improve gender translation and mitigate masculine bias (costajussà and de jorge, 2020; @xcite @xcite . to test these methods and inspect systems' behaviour, several template-based datasets have been made available -such as winomt @xcite or simplegen @xcite -which are especially intended to target occupational stereotyping. instead, natural datasets such as the arabic parallel gender corpus @xcite and gate @xcite allow for evaluation of gender bias under more naturalistic conditions. among such type corpora, must-she @xcite represents the only multilingual, natural test set designed to evaluate gender bias for both mt and st. already available for @xmath3, here we have contributed to its expansion for the @xmath4. as far as the topic of inclusivity and neutral language translation is concerned, research in mt is quite in its infancy. a notable exception is the work by @xcite , who created parallel test and fine-tuning data to develop mt systems able to generate non-binary translations for @xmath5. however, their target sentences are artificial -created by replacing gendered morphemes and articles with synthetic placeholders -thus serving only as a proofof-concept. @xcite , instead, are the first to advocate for the use of target genderneutral rephrasings and synonyms as a viable paradigm toward more inclusive mt when gender is unknown or simply irrelevant. @xcite and @xcite investigate the preservation of gender-neutral pronouns for korean/@xmath6. their results, however, show that current mt systems still face serious difficulties on relying on the inclusive, neutral pronoun they in translation. along this line of work, ines -to the best of our knowledge -represents the first test suite designed to asses the use of neutral, inclusive forms beside pronouns for translating into english. ## conclusion this paper summarizes the results of our wmt-2023 test suites evaluations, which focus on gender bias and inclusivity in translation. to this aim, we have introduced the en-de expansion of the multilingual must-she test set @xcite and the newly created ines dataset for deen. the former is designed to assess gender bias and translation across a qualitatively differentiated selection of feminine/masculine gender phenomena. ines, instead, measures systems' ability to generate inclusive english translations that do not rely on the use of masculine generics. results on must-she w m t 23 show that the evaluated mt systems are reasonably good at translating gender under realistic conditions, achieving comparable results across feminine and masculine gender translation. however, for ambiguous cases where the input sentence does not inform about the gender form to be used in translation, we confirm a strong skew where all systems tend to generate masculine forms almost by default. results on ines, instead, indicate that providing inclusive translations still represents a quite challenging task for current mt systems, in spite of the increasingly widespread use and preference for inclusive language forms in english. as a final remark, we acknowledge that the phenomena subject to our analysis (gender bias and gender inclusion) are not yet part of the repertoire of phenomena for which mt systems are currently designed. these systems are indeed primarily built with the goal of maximising translation quality in general rather than aspects of the problem, specifically fairness, for which sensitivity is still limited. all in all, however, this experience has allowed us to shed light on these issues, raise the awareness of the mt community and, hopefully, favour future developments.