Gender in Danger? Evaluating Speech Translation Technology on the M u ST - SHE Corpus (2020)

Abstract: Translating from languages without productive grammatical gender like English into gender-marked languages is a well-known difficulty for machines. This difficulty is also due to the fact that the training data on which models are built typically reflect the asymmetries of natural languages, gender bias included. Exclusively fed with textual data, machine translation is intrinsically constrained by the fact that the input sentence does not always contain clues about the gender identity of the referred human entities. But what happens with speech translation, where the input is an audio signal? Can audio provide additional information to reduce gender bias? We present the first thorough investigation of gender bias in speech translation, contributing with: i) the release of a benchmark useful for future studies, and ii) the comparison of different technologies (cascade and end-to-end) on two language directions (English-Italian/French).

## introduction with the exponential popularity of deep learning approaches for a great range of natural language processing (nlp) tasks being integrated in our daily life, the need to address the issues of gender fairness 1 and gender bias has become a growing interdisciplinary concern. present-day studies on a variety of nlp-related tasks, such as sentiment analysis @xcite coreference resolution @xcite @xcite , visual semantic-role labeling @xcite or language modeling @xcite , attest the existence of a systemic bias that reproduces gender stereotypes discriminating women. in translation-related tasks, gender bias arises from the extent through which each language formally expresses the female or male gender of a referred human entity. languages with a grammatical system of gender, such as romance languages, rely on a copious set of morphological (inflection) and syntactic (gender agreement) devices applying to numerous parts of speech @xcite . differently, english is a natural gender language that only reflects distinction of sex via pronouns, inherently gendered words (boy, girl) and exceptionally with marked nouns @xcite . for all the other indistinct neutral words, the gender of the referred entity -if available -is inferred from contextual information present in the discourse, e.g. he/she is a friend. nascent inquiries on machine translation (mt) pointed out that machines tend to reproduce the linguistic asymmetries present in the real-world data they are trained on. in the case of gender inequality, this is made apparent by the attribution of occupational roles from gender-neutral linguistic forms into marked ones, where mt often wrongly chooses male-denoting (pro)nouns, e.g. identifying scientist, engineer or doctor as men @xcite . failing to pick the appropriate feminine form is both a technical and an ethical matter: gender-related errors affect the accuracy of mt systems but, more significantly, a biased system can dangerously perpetuate the under-/misrepresentation of a demographic group @xcite . previous studies accounting for mt systems' strengths and weaknesses in the translation of gender shed light on the problem but, at the same time, have limitations. on one hand, the existing evaluations focused on gender bias were largely conducted on challenge datasets, which are controlled artificial benchmarks that provide a limited perspective on the extent of the phenomenon and may force unreliable conclusions @xcite @xcite . on the other hand, the natural corpora built on conversational language that were used in few studies @xcite include only a restricted quantity of not isolated gender-expressing forms, thus not permitting either extensive or targeted evaluations. moreover, no attempt has yet been made to assess if and how speech translation (st) systems are affected by this particular problem. as such, whether st technologies that leverage audio inputs can retrieve useful clues for translating gender in addition to contextual information present in the discourse, or supply for their lack, remains a largely unexplored question. in the light of above, the contributions of this paper are: on the two language pairs addressed, our comparative evaluation of cascade vs. end-to-end st systems indicates that the latter are able to better exploit audio information to translate specific gender phenomena, for which the cascade systems require externally-injected information. a second issue, particularly relevant to this research, is the information loss when passing from audio to text representations. even with perfect transcripts, subtle aspects that cannot be grasped from the text only (e.g. speaker's pitch as a clue of his/her gender) can only be reintroduced by injecting external knowledge to support the mt step @xcite . by avoiding intermediate text representations, direct end-to-end translation from audio to text @xcite can potentially cope with these limitations. however, due to the dearth of training corpora, it still underperforms with respect to the cascaded approach. recent evaluation campaigns @xcite have shown that, although the gap is gradually closing (less than 2 bleu points), cascade models still represent the state-of-the-art. in spite of the steady technological progress, little has so far been done to directly compare the two technologies on specific translation problems like the one addressed in this paper. measuring gender bias. previous attempts to test the production of gender-aware automatic translations solely focused on mt, where a widespread approach involves the creation of challenge datasets focused on specific linguistic phenomena. prates et al. (2018) and cho et al. (2019) construct template sentences using occupational or sentiment words associated with a gender-neutral pronoun, to be translated into an english gender-specified one ([x] is a professor: he/she is a professor). similarly, the occupations test @xcite and wino mt @xcite cast human entities into proto-or anti-stereotypical gender associations via coreference linking (e.g. the english sentence "the janitor does not like the baker because she/he always messes up the kitchen", where "the baker" is to be translated into spanish as la panadera or el panadero depending on the english pronoun). although such simple constructions allow for targeted experiments, artificial data characterized by a qualitatively limited variety of phenomena generate constrained environments that may produce biased results. as far as studies on naturally occurring data are concerned, @xcite estimate mt systems' performance in the realization of speaker's gender agreement on two male and female test sets containing first person singular pronouns. this strategy increases the chances to isolate speakerdependent gendered expressions, but still, the employed bleu metric does not pointedly grasp the effect of gender translation on the output, as the overall performance is also impacted by other factors. analogously, @xcite design a set of agreement rules to automatically recover 300 gender-affected sentences in their corpus, but the evaluation relies on global bleu scores computed on a bigger set (1,300 sentences) and does not consider male-female related differences. @xcite use a parser to detect morphological realizations of speakers' gender on a single femalespeaker corpus that does not permit inter-gender comparisons. in light of above, an ideal test set should consist of naturally occurring data exhibiting a diversified assortment of gender phenomena so to avoid forced predictions with over-controlled procedures. also, a consistent amount of equally distributed feminine and masculine gender realizations need to be identified to disentangle the accuracy of gender translation from the overall model's performance. accordingly, in §3 we present must-she, a multilingual test set designed for the investigation of gender bias in st, which, as explained in §4, is used for a targeted gender-sensitive evaluation approach. ## the must-she benchmark we built must-she on naturally occurring data retrieved from @xcite , the largest freely available multilingual corpus for st, which comprises (audio, transcript, translation) triplets extracted from ted talks data. besides being multilingual, must-c is characterized by high-quality speech and a variety of different speakers that adequately represent women, two aspects that determined its selection among other existing corpora @xcite @xcite . as such, must-she was compiled by targeting in the original dataset linguistic phenomena that entail a gender identification from english into italian and french, two romance languages that extensively express gender via feminine or masculine morphological markers on nouns, adjectives, verbs and other functional words (e.g. articles and demonstratives). ## conclusion if, like human beings, "machine learning is what it eats", the different "diet" of mt and st models can help them to develop different skills. one is the proper treatment of gender, a problem when translating from languages without productive grammatical gender into gender-marked ones. with respect to this problem, by eating parallel texts during training, mt performance is bounded by the statistical patterns learned from written material. by eating (audio, text) pairs, st has a potential advantage: the possibility to infer speakers' gender from input audio signals. we investigated for the first time the importance of this information in st, analysing the behaviour of cascade (the state of the art in the field) and end-to-end st technology (the emerging approach). to this aim, we created must-she, a benchmark annotated with different types of gender-related phenomena in two language directions. our evaluation shows that, in spite of lower overall performance, the direct approach can actually exploit audio information to better handle speaker-dependent gender phenomena. these are out of reach for cascade solutions, unless the mt step is supplied with external (not always accessible) knowledge about the speaker. back to our title: if, in st, gender is still in danger, we encourage our community to start its rescue from must-she and the findings discussed in this paper.