FBK @ IWSLT Test Suites Task: Gender Bias evaluation with M u ST - SHE (2024)

Abstract: This paper presents the FBK contribution to the IWSLT-2024 ‘Test suites’ shared subtask, part of the Offline Speech Translation Task. Our contribution consists of the MuST-SHE-IWSLT24 benchmark evaluation, designed to assess gender bias in speech translation. By focusing on the en-de language pair, we rely on a newly created test suite to investigate systems’ ability to correctly translate feminine and masculine gender. Our results indicate that – under realistic conditions – current ST systems achieve reasonable and comparable performance in correctly translating both feminine and masculine forms when contextual gender information is available. For ambiguous references to the speaker, however, we attest a consistent preference towards masculine gender, thus calling for future endeavours on the topic. Towards this goal we make MuST-SHE-IWSLT24 freely available at: https://mt.fbk.eu/must-she/

## introduction in today's interconnected world, speech translation technology stands as a cornerstone of global communication, facilitating seamless interactions across linguistic barriers. indeed, the last few years have seen notable advancements for the task of speech-to-text translation (st), which has made strides in generic performance @xcite @xcite . also, the emergence massively multilingual solutions has greatly expanded the language coverage of competitive "one-model-fits-all" speech models @xcite @xcite . amid such advancements, there arise the increasing need to pair traditional overall quality assessments of st with more fine-grained analyses by accounting for relevant aspects of translations. it is within this context that the iwslt test suites shared task emerges, aiming to provide a dedicated evaluation framework for specific dimensions of the st output, which are otherwise overlooked with generic test sets and holistic metrics. in light of the above, our contribution is dedicated to the critical themes of gender bias in automatic translation @xcite @xcite . 1 given the largescale deployment of st, biased translations are not only relevant from a technical perspective, where gender-related errors negatively impact the accuracy of automatic translation. rather, biased and non-inclusive systems can pose the concrete risk of under/misrepresenting gender minorities by overproducing masculine forms and reinforcing gendered stereotypes @xcite . indeed, gendered linguistic expressions affect the representation and perception of individuals @xcite @xcite , and are actively used as a tool to negotiate the social, personal, and political reality of gender @xcite . a such, models that systematically favor masculine over feminine forms fail to properly recognize women, can reduce feminine visibility, and offer an unequal service quality @xcite . this paper presents the fbk participation in the test suites shared task by conducting evaluations on the must-she iw slt 24 en-de dataset. it represents the newly created speech-to-text extension of the @xmath0, a multilingual gender bias benchmark @xcite . in the hereby presented evaluations, we obtained translations of our test suites by systems that are part of the offline speech translation task of the 21st international conference on spoken language @xcite , with annotated segments organized per category. for each gender-neutral word referring to a human entity in the english source sentence (src), the reference translation (ref) shows the corresponding gender-marked (fem/masc) forms, annotated with their @xmath1. the last column provides information about the speaker's gender. translation (iwslt 2024). specifically, we evaluated 13 systems for must-she iw slt 24 en-de. ## must-she iw slt 24 must-she iw slt 24 is a test suite designed to evaluate the ability of st systems to correctly translate gender. it is composed of 200 segments that require the translation of -at least -one english gender-neutral word into the corresponding masculine or feminine target word(s) in german. 2 the test suite is created as an extension of must-she, a multilingual, natural benchmark built on ted talks data @xcite . the original corpus comprises ∼3,000 (audio, transcript, translation) triplets annotated with qualitatively differentiated gender-related phenomena for thee language pairs: @xmath2. recently, must-she was also extended to @xmath3.e. must-she w m t 23 @xcite . however, since it only consists of a textual portion (transcript, translation), it does not allow for the evaluation of st models. here, we introduce the expansion of must-she @xmath4, by incorporating the additional speech input portion so as to obtain (audio, transcript, translation) triplets. ## results in starting from coverage scores (all-cov), all models achieve overall positive results, which range from ∼70% (hw-tsc_constrained-wllm.primary) to 74.79% (hw-tsc_constrained.primary). hence, these models produce a good amount of gender-marked words that can be evaluated with regards to the accuracy of their gender realization. moving onto the overall accuracy scores (all-acc), we can see that -while there is still room for improvement -all of the evaluated st systems achieve reasonable results, by being able to correctly translate gender with an accuracy of at least 77.42% (kit.contrastive1) up to 84.44% for hw-tsc_constrained.primary. similar accuracy ranges are attested also by disaggregating results across feminine (f-acc) and masculine (m-acc) genders. interestingly, such results show that none of the models exhibit perfectly equal performance across both genders. still, the divide is fairly limited, with i) a comparable number of st systems achieving slightly higher results on either the feminine or masculine set of must-she, and ii) little variation in scores across the 13 models, as attested in terms of standard deviation. if we go more fine-grained into disaggregated results, however, we unveil a higher degree of variation. ## conclusion this paper summarizes the results of our iwslt-2024 test suites evaluation, which focused on gender bias in translation. to this aim, we have introduced the speech expansion of the en-de must-she test set. overall, results on must-she iw slt 24 show that the evaluated st systems are reasonably good at translating gender under realistic conditions, achieving comparable results across feminine and masculine gender translation. also, all models are quite robust, and show a similar behaviour for translation of unambiguous gender phenomena, where they can rely on contextual gender information. however, for ambiguous cases where the input sentence does not inform about the gender form to be used in translation, we confirm a strong skew where all systems favour masculine generation almost by default. this finding calls for further research endeavours and evaluation initiatives to counter gender bias in st and measure future advances.