Hi Guys or Hi Folks? Benchmarking Gender-Neutral Machine Translation with the G e NTE Corpus (2023)

Abstract: Gender inequality is embedded in our communication practices and perpetuated in translation technologies. This becomes particularly apparent when translating into grammatical gender languages, where machine translation (MT) often defaults to masculine and stereotypical representations by making undue binary gender assumptions. Our work addresses the rising demand for inclusive language by focusing head-on on gender-neutral translation from English to Italian. We start from the essentials: proposing a dedicated benchmark and exploring automated evaluation methods. First, we introduce GeNTE, a natural, bilingual test set for gender-neutral translation, whose creation was informed by a survey on the perception and use of neutral language. Based on GeNTE, we then overview existing reference-based evaluation approaches, highlight their limits, and propose a reference-free method more suitable to assess gender-neutral translation.

## introduction societal gender asymmetries and inequalities are reflected and perpetuated through language @xcite . such awareness has grown also within the natural language processing (nlp) field @xcite , where extensive research has highlighted how several applications suffer from gender bias @xcite . as also noted by mt users themselves @xcite , among these applications are translation systems used at large scale, which pose the concrete risk of misrepresenting gender minorities by over-producing masculine forms, while reinforcing binary gendered expectations and stereotypes @xcite . to foster greater inclusivity and break free from the constraints of masculine/feminine language, * the authors contributed equally. neutral strategies have emerged and are increasingly adopted in academia (apa, 2020), institutions @xcite , and industry alike @xcite . these strategies aim to overcome marked forms that treat the masculine gender as the conceptually generic, default human prototype (e.g., humankind vs. mankind) @xcite . thus, they challenge gender norms and embrace all gender identities by avoiding gendered terms when unnecessary (e.g. chair vs. chairman/chairwoman) @xcite . english, being at the forefront of inclusive language changes and with its limited gendered grammar @xcite , has faced fewer obstacles in adapting to neutral forms, which have already been modeled into monolingual generative tasks @xcite . as recently underscored by @xcite , however, the resources and approaches made available for english are not portable to grammatical gender languages. such need for dedicated efforts is exemplified in italian, where neutral solutions must navigate the extensive encoding of masculine/feminine marking (e.g. the doctors are @xmath0: i/le dottori/esse sono qualificati/e) through synonymy or more complex rephrasing @xcite (e.g. → il personale medico [the medical staff]). while indeed more challenging, pursuing inclusivity in italian is relevant exactly because sexist attitudes are more visible and impactful in grammatical gender languages @xcite . nonetheless, the implementation of neutral language in mt remains to date a basically uncharted territory, despite the desirability of neutral outputs under several circumstances where gender is ambiguous or irrelevant. in light of the above, by focusing on @xmath1, we hereby lay the groundwork toward gender-neutral mt. starting from a survey aimed to understand the challenges of neutral translation in cross-lingual settings, we provide the necessary tools and resources to foster research on the topic by estimating gender neutral translation in mt. hence, our main contributions are: (1) a study on the feasibility of neutral translation, by surveying the potential trade-off among fluency, adequacy, and neutrality; (2) the creation of gente, foot_1 the first natural, parallel corpus designed to test mt systems' ability to generate neutral translations; (3) a comprehensive analysis of the (un)suitability of existing automatic metrics to evaluate neutral translation. as an inherent benchmark component, we indicate an alternative solution capable to better assess the task. we make the gente dataset freely available at https://mt.fbk.eu/gente/ and release the evaluation code under apache license 2.0 at https: //github.com/hlt-mt/fbk-neutr-eval . ## background emerging research has highlighted the importance of reshaping gender in nlp technologies in a more inclusive manner @xcite , also through the representation of non-binary identities and language @xcite @xcite . foundational works in this area have included several applications, such as coreference resolution systems (cao and @xcite , intra-lingual fair rewriters @xcite , and automatic classification of gender-neutral text @xcite . in mt, the research agenda has mainly focused on the improvement of masculine/feminine gender translation. along this line, different mitigation methods have been devised to ensure that unambiguous gendered referents (e.g. he/she is a doctor) are properly resolved in the target language (costajussà and de jorge, 2020; @xcite . these methods are often tested on synthetic template-based datasets such as winomt @xcite or simple-gen @xcite . as also stressed by @xcite , however, in realistic scenarios mt systems are also confronted with ambiguous input sentences that do not convey any gender distinction (e.g., en: i called the doctor). nonetheless, to date the resources and solutions envisioned for resolving such cases into grammatical gender languages like arabic @xcite , italian @xcite , spanish, or french @xcite entail offering two possible translation outputs, still constrained to binary gender forms (e.g., it: ho chiamato il dottore masc vs. la dottoressa fem). 2 as an exception within the current mt landscape, @xcite and @xcite investigate the preservation of genderambiguous pronouns for korean/@xmath2. since english can already boast the wellestablished neutral pronoun they, their study does not face the additional challenges of preserving such unmarked vagueness into grammatical gender languages. such challenges are exemplified by @xcite , who created parallel test and fine-tuning data to develop mt systems able to generate non-binary translations for @xmath3. however, their target sentences are artificial -created by replacing gendered morphemes and articles with synthetic placeholders -thus serving only as a proof-of-concept. to the best of our knowledge, @xcite are the first to advocate the use of target gender-neutral rephrasings and synonyms as a viable paradigm toward more inclusive mt when gender is unknown or simply irrelevant. despite this call to action, no concrete steps have been taken yet to actually facilitate research in this direction, not even toward suitable benchmarks to recognize the neutral forms occasionally generated by current systems @xcite . in light of the above, the path toward genderneutral translation in mt is bottlenecked by the lack of dedicated datasets and automated evaluations. here, we fill this gap so to guide and allow research on this novel topic. to this aim, we start in §3 by first ensuring that gender-neutral language can enable acceptable translations, not being perceived as inappropriate or intrusive. ## surveying gender-neutral translation neutralization is a form of linguistic gender inclusivity that relies on the retooling of established forms and grammar @xcite . according to the review of several gender-inclusive public guidelines by @xcite tives (e.g. il maestro vs. l'insegnante 3 ), to ii) more complex reformulations, which might involve altering the sentence structure (e.g. i miei colleghi vs. le persone con cui lavoro). 4 as such, to ensure neutrality, these solutions might have an effect in terms of brevity or perceived fluency. while widespread in monolingual, institutional contexts @xcite , the use of neutral forms in cross-lingual settings requires to weigh additional non-negligible factors. first, translations are bounded to a source text, whose meaning must be properly rendered in the target language. thus, more creative reformulations might collide with this instrinc constraint. also, it might be not always clear-cut when neutral translations ought to be performed. this is the case of masculine generics in the source language (e.g. all firemen): while they do not neatly fall under the idea of "ambiguous" input, their propagation to a target language clashes with the goal of inclusive mt itself. these issues stand unaddressed: the study by @xcite represents the only empirical investigation on the feasibility of genderneutral translation, but it is concerned with the cognitive effort that its realization poses to post-editors. therefore, to better understand the implications of gender-neutral translation for a wider range of stakeholders, we carried out a preliminary analysis on @xmath4. questionnaire. our survey was structured into two main parts. in part (i), we indirectly assessed linguistic acceptability: given a source english sentence paired with both a gendered (gt) and a neutral (nt) translation, we asked participants to indicate whether they had a preference or found 3 en: "the teacher". them to be equivalent foot_3 (see first, the linguistic acceptability of genderneutral translations was positively judged, perhaps at the higher rate than our own expectations. in fact, overall results indicate that in the majority of cases the nt was deemed preferable (42.5%) rather than equivalent to the gt (36.5%), where only a minority favoured gendered translations (21%). possibly, such trend is explained in light of participants' ideological preference for inclusive language rather than by purely linguistic factors. as shown in ## the gente corpus gente is the first test set designed to evaluate mt models' ability to perform gender-neutral translations, but only under desirable circumstances. in fact, when referents' gender is unknown or irrelevant, undue gender inferences should not be made and translation should be neutral. however, neutralization should not be always enforced; for instance, when a referent's gender is relevant and known, mt should not over-generalize to neutral translations. the corpus hence consists of 1,500 english-italian parallel sentences with mentions to human referents that equally represent two translation scenarios: 1) set-n, featuring genderambiguous source sentences that require to be neutrally rendered in translation; 2) set-g, featuring gender-unambiguous source sentences, which shall be properly rendered with gendered (masculine or feminine) forms in translation. altogether, these sets allow to benchmark whether systems are able to perform gender-neutral translation, and if they do so when appropriate. we build gente on naturally occurring instances of both scenarios retrieved from europarl @xcite . besides being a widely popular and high-quality mt resource, we chose this corpus inasmuch it represents formal communicative situations from the administrative/institutional domain. accordingly, it reflects the context for which gender-neutral forms are traditionally intended, also in line with the stakeholders' preference highlighted in §3. also, as examined by @xcite , europarl exhibits a large amount of genderambiguous cases that -although translated with gendered forms in the original references of the corpus -lend themselves as suitable candidates for neutralization. as explained in the forthcoming paragraphs ( §4.2), for each of these original europarl gendered target sentences, we create an additional gender-neutral reference translation. ## gender-neutral evaluation protocols we complement our benchmark creation effort with a study on the possible approaches for using gente to conduct automated evaluations of neutral mt. to this aim, we first define sound test-bed conditions ( §5.1). on this basis, we then experiment with a contrastive, reference-based protocol to inspect the effectiveness of standard mt metrics to assess neutral translation ( §5.2). then, to overcome the limitations encountered with the reference-based approach, we implement a reference-free protocol ( §5.3), which shows promise in advancing the task's evaluation. ## conclusions in this work, we investigated gender-neutral translation as a path for inclusive mt. to this aim, we focused on @xmath5, a pair that is highly representative of the challenges of implementing neutral forms into grammatical gender languages. as a novel area of inquiry, we started from the fundamentals. first, we conducted a survey on the acceptability of gender-neutral translation, which highlighted the openness of potential mt end-users, especially in formal communicative situations. second, informed by the survey, we built gente, the first natural benchmark for evaluating genderneutral translation in mt. third, we investigated the (un)suitability of existing automatic evaluation protocols to assess gender-neutral translation, and thus proposing an alternative, reference-free solution. having taken the first steps toward genderneutral mt, our resources and evaluation method are made available to foster and inform the future development of more inclusive mt.