Overview of the Shared Task on Machine Translation Gender Bias Evaluation with Multilingual Holistic Bias (2024)

Abstract: We describe the details of the Shared Task of the 5th ACL Workshop on Gender Bias in Natural Language Processing (GeBNLP 2024). The task uses dataset to investigate the quality of Machine Translation systems on a particular case of gender robustness. We report baseline results as well as the results of the first participants. The shared task will be permanently available in the Dynabench platform.

## introduction gender bias poses challenges across various aspects of automatic translation. these challenges include preserving correct pronouns, understanding the correct gendered context, and relating adjectives and professions to the proper gender. the issue becomes even more complex when considering multilingual translation, especially for lowresource languages. the gebnlp 2024 workshop aims to raise awareness of these challenges by introducing a dedicated shared task for investigating translation quality using the multilingual holis-ticbias dataset @xcite . this initiative seeks to foster a community-driven effort and long-term solutions toward improving gender representation in machine translation. we encourage researchers to contribute their expertise, not just for the workshop but for the ongoing pursuit of advancements in this field. ## motivation the development of gender @xcite @xcite @xcite @xcite or demographic-specific @xcite datasets has raised the interest in evaluating natural language processing (nlp) models beyond standard quality terms. in machine translation (mt), gender bias is observed when translations show errors in linguistic gender determination despite the fact that there are sufficient gender clues in the source content for a system to infer the correct gendered forms. to illustrate this phenomenon, sentence (1) in mi amigo es un amo de casa. gender bias is observed when an mt system produces the wrong gendered form when translating sentence (2) into a language that uses distinct gendered forms for the word doctor. on the contrary, a single error in the translation of an utterance such as sentence (1) would not be sufficient to conclude that gender bias exists in the model; doing so would take consistently observing one linguistic gender over another. finally, a lack of robustness would be shown if the translation quality differed for the two sentences in (3). it has previously been hypothesized that one possible source of gender bias in mt is gender representation imbalance in large training and evaluation data sets, e.g., @xcite ; @xcite . our task goes beyond previous gender bias mt evaluation efforts, such as @xcite al. (2019); renduchintala et al. (2021); levy et al. (2021); costa-jussà et al. (2022); renduchintala and williams (2022); savoldi et al. (2021); alhafni et al. (2022); attanasio et al. (2023) , to name a few, mainly by increasing the number of languages and fairly comparing three main gender mt issues which are gender-specific, gender robustness, and unambiguous gender (see section 4). ## goals the goals of the multilingual holisticbias task as part of the 5th acl workshop on gender bias in natural language processing are: ## multilingual holisticbias task we propose to evaluate three cases of gender bias: gender-specific, gender robustness, and unambiguous gender translation. ## results this section reports results for the two baselines and the submitted system. we provide results only for task 2 (gender robustness), which received the submission. finally, ## final remarks this paper introduces the multilingual holisticbias dynabench task foot_0 which has been launched in the context of the 5th acl worskhop on gender bias in nlp foot_1 . this task will remain open for participation. at the moment of the preparation of this paper, we have received a single participation which evaluates the mitigation strategy of dama @xcite for gender robustness. we are also reporting strong baseline results with nllb models for this particular task. however, we do not include baselines for gender-specification and unambiguous gender, which is left as further work. we are looking forward to receiving more submissions in the near future. also notice that an extension of the multilingual holisticbias dataset is currently going on and released @xcite .