Breeding Gender-aware Direct Speech Translation Systems (2020)

Abstract: In automatic speech translation (ST), traditional cascade approaches involving separate transcription and translation steps are giving ground to increasingly competitive and more robust direct solutions. In particular, by translating speech audio data without intermediate transcription, direct ST models are able to leverage and preserve essential information present in the input (e.g.speaker’s vocal characteristics) that is otherwise lost in the cascade framework. Although such ability proved to be useful for gender translation, direct ST is nonetheless affected by gender bias just like its cascade counterpart, as well as machine translation and numerous other natural language processing applications. Moreover, direct ST systems that exclusively rely on vocal biometric features as a gender cue can be unsuitable or even potentially problematic for certain users. Going beyond speech signals, in this paper we compare different approaches to inform direct ST models about the speaker’s gender and test their ability to handle gender translation from English into Italian and French. To this aim, we manually annotated large datasets with speak-ers’ gender information and used them for experiments reflecting different possible real-world scenarios. Our results show that gender-aware direct ST solutions can significantly outperform strong – but gender-unaware – direct ST models. In particular, the translation of gender-marked words can increase up to 30 points in accuracy while preserving overall translation quality.

## introduction language use is intrinsically social and situated as it varies across groups and even individuals @xcite . as a result, the language data that are collected to build the corpora on which natural language processing models are trained are often far from being homogeneous and rarely offer a fair representation of different demographic groups and their linguistic behaviours @xcite . consequently, as predictive models learn from the data distribution they have seen, they tend to favor the demographic group most represented in their training data @xcite . this brings serious social consequences as well, since the people who are more likely to be underrepresented within datasets are those whose representation is often less accounted for within our society. a case in point regards the gender data gap. 1 in fact, studies on speech taggers @xcite and speech recognition @xcite showed that the underrepresentation of female speakers in the training data leads to significantly lower accuracy in modeling that demographic group. the problem of gender-related differences has also been inspected within automatic translation, both from text @xcite and from audio @xcite . these studies -focused on the translation of spoken language -revealed a systemic gender bias whenever systems are required to overtly and formally express speaker's gender in the target languages, while translating from languages that do not convey such information. indeed, languages with grammatical gender, such as french and † the authors contributed equally. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . italian, display a complex morphosyntactic and semantic system of gender agreement @xcite , relying on feminine/masculine markings that reflect speakers' gender on numerous parts of speech whenever they are talking about themselves (e.g. en: i've never been there -it: non ci sono mai stata/stato). differently, english is a natural gender language @xcite that mostly conveys gender via its pronoun system, but only for third-person pronouns (he/she), thus to refer to an entity other than the speaker. as the example shows, in absence of contextual information (e.g as a woman, i have never been there) correctly translating gender can be prohibitive. this is the case of traditional text-to-text machine translation (mt) and of the so-called cascade approaches to speech-to-text translation (st), which involve separate transcription and translation steps @xcite . instead, direct approaches @xcite translate without intermediate transcriptions. although this makes them partially capable of extracting useful information from the input (e.g. by inferring speaker's gender from his/her vocal characteristics), the general problem persists: since female speakers (and associated feminine marked words) are less frequent within the training corpora, automatic translation tends towards a masculine default. following @xcite , this attested systemic bias can directly affect the users of such technology by diminishing their gender identity or further exacerbating existing social inequalities and access to opportunities for women. systematic gender representation problems -although unintended -can affect users' self-esteem @xcite , especially when the linguistic bias is shaped as a perpetuation of stereotypical gender roles and associations @xcite . additionally, as the system does not perform equally well across gender groups, such tools may not be suitable for women, excluding them from benefiting from new technological resources. to date, few attempts have been made towards developing gender-aware translation models, and surprisingly, almost exclusively within the mt community @xcite @xcite . the only work on gender bias in st @xcite proved that direct st has an advantage when it comes to speaker-dependent gender translation (as in i've never been there uttered by a woman), since it can leverage acoustic properties from the audio input (e.g. speaker's fundamental frequency). however, relying on perceptual markers of speakers' gender is not the best solution for all kinds of users (e.g. transgenders, children, vocally-impaired people). moreover, although their conclusions remark that direct st is nonetheless affected by gender bias, no attempt has yet been made to try and enhance its gender translation capability. following these observations, and considering that st applications have entered widespread societal use, we believe that more effort should be put into further investigating and controlling gender translation in direct st, in particular when the gender of the speaker is known in advance. towards this objective, we annotated @xcite ) -the largest freely available multilingual corpus for st -with speakers' gender information and explored different techniques to exploit such information in direct st. the proposed techniques are compared, both in terms of overall translation quality as well as accuracy in the translation of gender-marked words, against a "pure" model that solely relies on the speakers' vocal characteristics for gender disambiguation. in light of the above, our contributions are: (1) the manual annotation of the ted talks contained in must-c with speakers' gender information, based on the personal pronouns found in their ted profile. the resource is released under a cc by nc nd 4.0 international license, and is freely downloadable at https://ict.fbk.eu/ must-speakers/; experiments carried out on english-italian and english-french show that, on both language directions, our gender-aware systems significantly outperform "pure" st models in the translation of gender-marked words (up to 30 points in accuracy) while preserving overall translation quality. moreover, our best systems learn to produce feminine/masculine gender forms regardless of the perceptual features received from the audio signal, offering a solution for cases where relying on speakers' vocal characteristics is detrimental to a proper gender translation. ## background besides the abundant work carried out for english monolingual nlp tasks @xcite , a consistent amount of studies have now inspected how mt is affected by the problem of gender bias. most of them, however, do not focus on speaker-dependent gender agreement. rather, a number of studies @xcite @xcite ) evaluate whether mt is able to associate prononimal coreference with an occupational noun to produce the correct masculine/feminine forms in the target gender-inflected languages (en: i've known her for a long time, my friend is a cook. es: la conozco desde hace mucho tiempo, mi amiga es cocinera). notably, few approaches have been employed to make neural mt systems speaker-aware by controlling gender realization in their output. @xcite enrich their data with a set of genderagreement rules so to force the system to account for them in the prediction step. in @xcite , the mt system is augmented at training time by prepending a gender token (female or male) to each source segment. similarly, @xcite artificially inject a short phrase (e.g. she said) at inference time, which acts as a gender domain label for the entire sentence. these approaches are implemented and tested on natural spoken language that, compared to written language, is more likely to contain references to the speaker and, consequently, speaker-dependent gender-marked words. in the light of above, the correct translation of gender is a particularly relevant task for st systems, as they are precisely developed to translate oral, conversational language. nonetheless, to our knowledge only one work has investigated gender bias in st @xcite . focusing on the proper handling of gender phenomena, the authors take stock of the situation by comparing cascade and direct architectures on must-she, a multilingual benchmark derived from the ted-based must-c corpus and specifically designed to evaluate gender translation and bias in st. their conclusions remark that, although traditional cascade systems still outperform direct solutions, the latter are able to exploit audio information for a better treatment of speaker-dependent gender phenomena. these findings open a line of focused research on speaker-aware st that is worth exploring more thoroughly, also in light of the fact that the performance gap between cascade and direct approaches has further reduced @xcite . on one side, rather than comparing the two paradigms, this progress now motivates exploring all the possible ways to boost direct st performance towards the translation of gender-marked expressions. on the other side, since the direct systems tested in @xcite rely on "pure" models built to verify an hypothesis (i.e. that translating audio signals without intermediate representations makes a difference in handling gender), the real potential of direct st technology with respect to this problem is still unknown. moreover, as their "pure" models solely rely on the speaker's fundamental frequency, various instances in which such perceptual marker is not indicative of the speaker's gender remain out of the picture. ## annotation of must-c with speakers' gender information although current research on gender-aware st can count on the must-she benchmark @xcite for fine-grained evaluations, gender-annotated training data are not yet available. so far, this has limited the scope of research to application scenarios in which speakers' gender is inferred from the input audio. these scenarios are not representative of the full range of possible usages of st and are also potentially problematic, since gendered forms expected in translation do not necessarily align with speaker's vocal characteristics. in the light of the above, building large training corpora explicitly annotated with gender information becomes crucial. to this aim, rather than building a new resource from scratch, we opted for adding an annotation layer to must-c, which has been chosen over other existing corpora (iranzo-sánchez et al., 2020) for the following reasons: i) it is currently the largest freely available multilingual corpus for st, ii) being based on ted talks it is the most compatible one with must-she, iii) ted speakers' personal information is publicly available and retrievable on the ted official website. 2 following the must-c talk ids, we have been able to i) automatically retrieve the speakers' name, talks m talks f hours m hours f segments m segments f en-it 1,569 725 316 136 178,841 71,877 en-fr 1,569 725 327 151 189,742 81, 527 differently, following the guidelines in @xcite , we do not want to run the risk of making assumptions about speakers' gender identity and introducing additional bias within an environment that has been specifically designed to inspect gender bias. by looking at the personal pronouns used by the speakers to describe themselves, our manual assignment instead is meant to account for the gender linguistic forms by which the speakers accept to be referred to in english @xcite , and would want their translations to conform to. we stress that gendered linguistic expressions do not directly map to speakers' self-determined gender identity @xcite . we therefore make explicit that throughout the paper, when talking about speakers' gender, we refer to their accepted linguistic expression of gender rather than their gender identity. focusing on the two language pairs of our interest, 2,294 different speakers described via he/she pronouns foot_1 are represented in both en-it and en-fr. their male/female foot_2 distribution is unbalanced, as shown in ## st systems for our experiments, we built three types of direct systems. one is the base system, a state-of-the-art model that does not leverage any external information about speaker's gender ( §4.1). the others are two gender-aware systems that exploit speakers' gender information in different ways: multi-gender ( §4.2) and specialized ( §4.3). all the models share the same architecture, a transformer @xcite adapted to st. the encoder processes the input mel-filter-bank sequences with two 2d convolutional layers with stride 2, returning a sequence that is four times shorter than the original input. the vectors of this sequence are projected by a linear transformation into the dimensional space used in the following encoder transformer layers and are summed with sinusoidal positional embeddings. the attentions in the encoder layers are biased toward elements close on the time dimension with a logarithmic distance penalty @xcite . the decoder architecture, instead, is not modified. ## manual analysis we complement our automatic evaluation with a manual inspection on the output of three models: base, multi-decprep-all (multi), and specialized-all (spec). for each model, we analyzed the translation of 100 common segments across en-it/en-fr, which allow for cross-lingual comparisons. we first take into account those instances where systems' accuracy in the production of gender-marked words was measurable, as in (a), (b), (c) in overall, (a), (b) and (c) clearly attest the progressively improved performance from base to multi and spec. in particular, in (c), spec is able to pick the required masculine form in spite of a contextual hint about a second female referent (woman), thus overcoming what is a difficult prediction even for multi. we also inspect those cases where systems' accuracy on gender production was not measurable to cast some light on the reasons for a limited term coverge. we found that, while there are some generally wrong translations -(d) -such instances only amount to 1/3 of the cases. in the remaining 2/3, the output is fluent and reflects the source utterance meaning but it simply does not match the exact annotated word in the reference. we found that st translations often offer alternative constructions that do not require an overt gender-inflection -(e) -or rely on appropriate gender-marked synonyms of the word in the reference -(f). we can hence conclude that many gender translations that do not contribute to gender accuracy confirm an improved ability of the enriched models in gender translation. ## conclusion we rose to the challenge posed by @xcite to further explore gender translation in direct st. going beyond direct systems' attested ability to leverage speaker's vocal characteristics from the audio input, we developed gender-aware models suitable for operating conditions where speaker's gender is known. to this aim, we annotated the large must-c dataset with speaker's gender information, and used the new annotations to experiment with different architectural solutions: "multi-gender" and "specialized". our results on two language pairs (en-it and en-fr) show that breeding speaker's gender-aware st improves the correct realization of gender. in particular, our specialized systems outperform the gender-unaware st models by 30 points in gender accuracy without affecting overall translation quality.