Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation (2022)

Abstract: Gender bias is largely recognized as a problematic phenomenon affecting language technologies, with recent studies underscoring that it might surface differently across languages. However, most of current evaluation practices adopt a word-level focus on a narrow set of occupational nouns under synthetic conditions. Such protocols overlook key features of grammatical gender languages, which are characterized by morphosyntactic chains of gender agreement, marked on a variety of lexical items and parts-of-speech (POS). To overcome this limitation, we enrich the natural, gender-sensitive MuST-SHE corpus (Bentivogli et al., 2020) with two new linguistic annotation layers (POS and agreement chains), and explore to what extent different lexical categories and agreement phenomena are impacted by gender skews. Focusing on speech translation, we conduct a multifaceted evaluation on three language directions (English-French/Italian/Spanish), with models trained on varying amounts of data and different word segmentation techniques. By shedding light on model behaviours, gender bias, and its detection at several levels of granularity, our findings emphasize the value of dedicated analyses beyond aggregated overall results.

## introduction as @xcite posits: "gender is perhaps the only grammatical category that ever evoked passion -and not only among linguists." that is because, in the case of human entities, masculine or feminine inflections are assigned semantically, i.e. in relation to the extra-linguistic reality of gender @xcite @xcite . thus, gendered features interact with the -sociocultural and political -perception and representation of individuals @xcite , by prompting discussions on the appropriate recognition of gender groups and their linguistic visibility @xcite @xcite . such concerns also invested language technologies @xcite , where it has been shown that automatic translation systems tend to over-represent masculine forms and amplify stereotypes when translating into grammatical gender languages @xcite . current evaluation practices for assessing gender bias in both machine (mt) and speech translation (st) commonly inspect such concerning behaviours by focusing only on a restricted set of occupational nouns (e.g. nurse, doctor), and on synthetic benchmarks @xcite @xcite . also, even when relying on lexically richer natural benchmarks, the designed metrics still work at the word level, treating all gender-marked words indiscriminately @xcite . accordingly, current test sets and protocols: i) do not allow us to inspect if and to what extent different word categories participate in gender bias, ii) overlook the underlying morphosyntactic nature of grammatical gender on agreement chains, which cannot be monitored on single isolated words (e.g. en: a strange friend; it: una/o strana/o amica/o). in fact, to be grammatically correct, each word in the chain has to be inflected with the same (masculine or feminine) gender form. 1 we believe that fine-grained evaluations including the analysis of gender agreement across different parts of speech (pos) are relevant not only to gain a deeper understanding of bias in grammatical gender languages, but also to inform mitigating strategies and data curation procedures. toward these goals, our contributions are as follows. (1) we enrich must-she @xcite -the only natural gender-sensitive benchmark available for mt and also st -with two layers of linguistic information: pos and agreement chains. 2 (2) in light of recent studies exploring how model design and overall perfomance interplay with gender bias @xcite , we rely on our manually curated resource to compare three st models, which are trained on varying amounts of data, and built with different segmentation techniques: character and byte-pairencoding (bpe) @xcite . we carry out a multifaceted evaluation that includes automatic and extensive manual analyses on three language pairs (en-es, en-fr, en-it) and we consistently find that: i) not all pos are equally impacted by gender bias; ii) translating words in agreement does not emerge as a systematic issue; iii) st systems produce a considerable amount of neutral rewordings in lieu of gender-marked expressions, which current binary benchmarks fail to recognize. finally, in line with concurring studies, we find that iv) character-based systems have an edge on translating gender phenomena, by favouring morphological and lexical diversity. ## background while research in natural language processing (nlp) initially prioritized narrow technical interventions to address the social impact of language technologies, we are recently attesting a shift toward a more comprehensive understanding of bias @xcite . along this line, focus has been given to bias analysis in models' innards and outputs @xcite , and to ascertain the validity of bias measurement practices @xcite @xcite . complementary mounting evidence suggests that -rather than striving for generalizations -gender bias detection ought to incorporate contextual and linguistic specificity @xcite @xcite @xcite , which however receives little attention due to a heavy focus on english nlp @xcite . purported agnostic approaches and evaluations @xcite can prevent from drawing reliable conclusions and mitigating recommendations, as attested by monolingual studies on grammatical gender languages @xcite @xcite and in figure 1 : example of gender-mapping in translation from the parallel en-it portion of the natural must-she corpus. unlike english, where gender is only expressed on few lexical and pronominal items (she, woman), in a grammatical gender language like italian, gender inflections (here feminine -a) are expressed on several linguistic items (e.g. verb-sentita, adjectivesicura) that are in agreement. unlike english, grammatical gender languages exhibit an elaborate morphological and syntactic system, where gender is overtly marked on numerous pos (e.g., verbs, determiners, nouns), and related words have to agree on the same gender features (see figure 1 for an example). still, current corpora and evaluation practices do not fully foreground systems' behaviour on such grammatical constraints. winomt @xcite represents the standard corpus to evaluate gender bias in mt within an english-to-grammatical gender language scenario. it has been progressively enriched with new features @xcite , and adapted for st @xcite . while this resource can be useful to diagnose gender stereotyping at scale, it excludes languages' peculiarities since it is built on the concatenation of two corpora designed for english monolingual tasks foot_2 -winogender @xcite and winobias @xcite -which consist of synthetic sentences with the same structure and a pre-selected occupational lexicon (e.g. "the lawyer yelled at the hairdresser because he did a bad job"). 4 to increase variability, troles and schmid (2021) extend winobias by accompanying occupations with highly gender-stereotypical verbs and adjectives. their evaluation though, still only considers the translated professions as to verify if the co-occuring words might skew the models' assumptions. however, gender-marking involves also several other, so far less accounted pos categories, but if they are just as problematic is not clear yet. existing bilingual @xcite , and multilingual @xcite natural benchmarks, instead, are manually curated as to identify a variety of gender phenomena specifically modeled on the accounted languages. as a result, they maximize lexical and contextual variability to inspect whether translation models yield feminine under-representation in real-world-like scenarios @xcite . however, since this variability is not mapped into fine-grained linguistic information, evaluations on such corpora do not single out which instances may be more responsible for gender bias. finally, by considering each word in isolation, they neglect the underlying features of gender agreement, which determine the grammatical acceptability of the translation. to the best of our knowledge, only two works have currently interplayed issues of syntactic agreement and gender bias. @xcite designed a set of english sentences involving a syntactic construction that requires to translate an occupational term according to its unequivocal "gender trigger" (e.g. that nurse is a funny man). while they find that mt struggles even in such a simple setting, they only inspect the translation of a single disambiguated word (nurse) rather than a whole group of words in agreement. closer to our intent, @xcite analyze the output of different st systems and note that their models seem to wrongly pick divergent gender inflections for unrelated words in the same sentence (e.g. en: as a researcher, professor; fr: en tant que chercheuse f , professeur m ) but not for dependency-related ones (e.g. en: the classic asian student; it: [la classica studentessa asiatica] f ). although limited in scope, their observation is worth being explored systematically. we thus conduct the very first study that intersects pos, agreement, and gender bias. ## must-she enrichment in light of the above, a fine-grained evaluation of bias focused on pos and gender agreement requires the creation of a new dedicated resource. rather than building it from scratch, we add two annotation layers to the existing must-she bench- parts-of-speech (a) src as one of the first women... ref f r en tant que l'une p ron des premières adj-det femmes.. (b) src as a child growing up in nigeria... refit da bambino n oun cresciuto v erb in nigeria. (c) src then an amazing colleague... refes luego una art asombrosa adj-des colega... agreement (d) src i was the first muslim homecoming queen, the first somali student senator... refes fui [la primera reina musulmana] del baile, [la primera senadora] somalí estudiantil... (e) src she's also been interested in research. refit e' [stata anche attratta] dalla ricerca . (f) src i also became a high school teacher. ref f r je suis aussi [devenu un professeur] de lycée. in the reference translations of the corpus, each target gender-marked word -corresponding to a neutral expression in the english source -is annotated with its alternative wrong gender form (e.g. en: the girl left; it: @xmath1. as further discussed in in section 4.2, such a feature enables fine-grained analyses of gender realization, which can also disentangle systems' tendency to (over)generate masculine -over feminine forms -in translation. must-she thus allows the identification and pinpointed evaluation of numerous and qualitatively different grammatical gender instances under authentic conditions. furthermore, the target languages covered in must-she (es, fr, it) are particularly suitable to focus on linguistic specificity. as a matter of fact, as @xcite suggest, accounting for gender in languages with similar typological features allows for proper comparisons. 6 3.1 phenomena categorization parts-of-speech. we annotate each target gender-marked word in must-she with pos information. as shown in agreement. we also enrich must-she with linguistic information that is relevant to investigate the morphosyntactic nature of grammatical gender agreement. gender agreement, or concord @xcite , requires that related words match the same gender form, as in the case of phrases, i.e. groups of words that constitute a single linguistic unit. 8 thus, as shown in ## gender agreement evaluation 6.1 automatic analysis the final step in our multifaceted analysis goes beyond the word level to inspect agreement chains in translation. to this aim, we define coverage as the proportion of generated chains matching with those annotated in must-she. then, the accuracy of the generated chains accounts for 3 different cases where: i) agreement is respected, and with the correct gender (c); ii) agreement is respected, but with the wrong gender (w); and iii) both feminine and masculine gender inflections occur together, and thus agreement is not respected (no). ## conclusion the complex system of grammatical gender languages entails several morphosyntactic implications for different lexical categories. in this paper, we underscored such implications and explored how different pos and grammatical agreement are involved in gender bias. to this aim, we enriched the must-she benchmark with new linguistic information, and carried out an extensive evaluation on the behaviour of st models built with different segmentation techniques and data quantities. on three language pairs (english-french/italian/spanish), our study shows that, while all pos are subject to masculine skews, they are not impacted to the same extent. respecting gender agreement for the translation of related words, instead, is not an issue for current st models. we also find that st generates a considerable amount of neutral expressions, suitable to replace gender-inflected ones, which however current test sets do not recognize. overall, our work reiterates the importance of dedicated analyses that, unlike holistic metrics, can single out system's behaviour on gender phenomena. accordingly, our results are in line with previous studies showing that, in spite of lower generic performance, character-based segmentation exhibits a better capability at handling feminine translation at different levels of granularity. as our must-she extension is available for both st and mt, we invite mt studies to start from our discoveries and resource. direct st technologies. also, our results disaggregated by pos invite reflections on how to intend and mitigate bias by means of interventions on the training data. in fact, while it is known that the must-c corpus @xcite used for training comprises a majority of masculine speakers, 21 the fact that certain lexical categories are more biased than others suggests that, on top of more coarse-grained quantitative attempts at gender balancing (costa-jussà and de jorge, 2020), data curation ought to account for more sensitive, nuanced, and qualitative asymmetries. these also imply how, rather than only how often, gender groups are represented @xcite . also, while nouns come forth as the most problematic pos, current practices of data augmentation based on a pre-defined occupational lexicon may address stereotyping @xcite , but do not increase the production of other nonetheless skewed lexical categories. overall, our enriched resource foot_18 can be useful to monitor the validity of different technical interventions. ethic statement. the use of gender as a variable @xcite warrants some ethical reflections. our evaluation on the must-she benchmark exclusively accounts for linguistic gender expressions. as reported in must-she data statement @xcite , 23 also for the subset of sentences that contain first-person references 24 (e.g. i'm a student), speakers' gender information is manually annotated based on the personal pronouns found in their publicly available personal ted profile, and used to check that the indicated (english) linguistic gender forms are rendered in the gold standard translations. while our experiments are limited to the binary linguistic forms represented in the used data, to the best of our knowledge, st natural language corpora going beyond binarism do not yet exist. 25 this is also due to the fact that unlike english -which finds itself for several cultural and linguistic reasons as a leader of change toward inclusive forms @xcite -direct nonbinary language based on neomorphemes (shroy, 2016; papadopoulos, 2019; knisely, 2020) is nontrivial to fully implement in grammatical gender languages @xcite and still object of experimentation @xcite . however, our manual evaluation expands to the possibility of inl strategies that could be detected in system's output. we underscore that such strategies are recommended and fruitful to avoid the gendering of referents, but are to be considered as concurring to -rather than replacements ofemerging linguistic innovations @xcite . lastly, we signal that direct st models may leverage speakers' vocal characteristics as a gender cue to infer gender translation. although the potential risks of such condition do not emerge and are not addressed in our setting (focused on pos and agreement features as a variable), we endorse the point made by @xcite . namely, direct st systems leveraging speaker's vocal biometric features as a gender cue can entail real-world dangers, like the categorization of individuals by means of biological essentialist frameworks (zimman, 2020). this can reduce gender to stereotypical expectations about how masculine or feminine voices should sound, and can be especially harmful to transgender individuals, as it can lead to misgendering @xcite and invalidation. note that we experimented with unmodified models for the sake of hypothesis testing without adding variability, but real-world deployment of st technologies must account for the potential harms arising form the use of direct st technologies as is.