g EN der- IT : An Annotated E nglish- I talian Parallel Challenge Set for Cross-Linguistic Natural Gender Phenomena (2021)

Abstract: Languages differ in terms of the absence or presence of gender features, the number of gender classes and whether and where gender features are explicitly marked. These cross-linguistic differences can lead to ambiguities that are difficult to resolve, especially for sentence-level MT systems. The identification of ambiguity and its subsequent resolution is a challenging task for which currently there aren’t any specific resources or challenge sets available. In this paper, we introduce gENder-IT, an English–Italian challenge set focusing on the resolution of natural gender phenomena by providing word-level gender tags on the English source side and multiple gender alternative translations, where needed, on the Italian target side.

## introduction cross-linguistic differences between languages often require implicit information in the source to be made explicit on the target side. when faced with systematic structural differences between the source and target languages, human translators rely on the (broader) context (linguistic, extralinguistic, world-knowledge) in order to infer the necessary information and adapt the target side accordingly. one such way in which many languages systematically differ is in terms of grammatical gender. languages not only differ in terms of the absence or presence of specific gender features but also in the number of (linguistic) gender classes, how and where gender features are marked, and in the underlying rules by which gender is assigned @xcite . 1 in languages with grammatical gender all nouns have an (arbitrarily) assigned lexical gender. 2 in most cases, the lexical gender of a noun is covert and can only be inferred from the morphological agreement with other words (articles, verbs, adjectives...) @xcite . however, when nouns refer to animate referents, overt gender markings corresponding to the so-called 'natural gender' (biological sex) of the referent are common (e.g. the spanish word for 'nurse' is 'enfermero' (male) or 'enfermera' (female)). such forms are generated using derivational suffixes and are often derived from the 'generic male'. this process is sometimes denoted as 'female marking' @xcite . while language learners encounter difficulties memorizing the lexically stored gender of foreign nouns @xcite , machine translation (mt) technology, given the limited (linguistic and extralinguistic) context most mt tools leverage, struggles with the explicitation of ambiguous forms, i.e. the process of disambiguation. so far, little research has been conducted on controlling the output of mt systems in terms of features such as gender and/or number that arise due to specific cross-linguistic differences. we believe that there are two main reasons for this: (i) the research that has been conducted in this area shows that controlling specific features is a technically very challenging problem. especially given the fact that it often requires in-depth linguistic knowledge and specialized linguistic tools, the performance of the latter often depending on how well-researched and fusing as it is often used as a synonym for noun class. there are, for instance, language with more than 3 gender classes (e.g. kiswahili has 9) as the classes are based on different semantic distinctions. likewise, there are languags with only two 'gender classes' which correspond e.g. to an animate vs inanimate distinction. 2 the dutch word for 'sun' is 'zon' (female), while the french word for 'sun' is 'soleil' (male). well-resourced the languages in question are; (ii) the lack of high-quality, human-crafted challenge sets that target specific cross-linguistic phenomena. in this paper, we present a word-level (human) annotated, adapted and cleaned version of a subset (english-italian) of the must-she corpus @xcite . 3 the main contribution of our work is threefold: (i) the must-she corpus focuses solely on gender agreement with the first person singular. our extension provides simple word-level annotations for all nouns and pronouns referring to human referents for the english sentences; (ii) while the transcripts of must-she are accompanied with gender information (male, female) of the speaker on a short paragraph-level, our word-level tags can be either male or female, but also ambiguous, when the sentence itself does not provide any explicit clues with respect to the gender of the referents; (iii) we focus on the textually gender-ambiguous sentences and provide all the correct gender-alternative translations for italian. the main motivations behind our work are the following: (i) first of all, there is a need for controlled diversity within the field of mt when it comes to controlling specific features of translations, specifically when dealing with structural cross-linguistic differences (vanmassenhove, 2020). to allow for controlled diversity, we created the first test set that allows research on identifying ambiguity and generating multiple translation variants in terms of gender; (ii) second, recent work by @xcite indicates that even a (very) small synthetic set of high quality sentences annotated for gender can be leveraged to improve the accuracy of translations in terms of gender specific phenomena without decreasing the overall quality. their work was limited to annotations for one referent per synthetic sentence and focused specifically on debiasing data in terms of gender. as highlighted in @xcite and saunders (2020), the effects of specific interventions need to be carefully examined on test sets that capture the complexity of a problem to its full extent. the manually annotated test set created does so by relying on 'natural' (as opposed to synthetic) data that is not limited to a single human referent per sentence. bias statement @xcite in summary, this dataset is intended to encourage work on gender bias in mt, but could equally be leveraged for monolingual research on the generation of gender diverse translations (in italian) and gender identification of referents (for english). the detailed analysis on english-italian is intended to raise awareness on cross-linguistic differences between languages in terms of gender. nlp technologies are prone to the perpetuation (and possibly also the exacerbation (vanmassenhove et al., 2021)) of inappropriate stereotypes and are currently unable to recognize or warn the user about the (gender) assumptions that have been made (e.g. by translating ambiguous source sentences systematically into one specific gendered variant on the target side). furthermore, current systems lack the ability for the user to indicate and/or control the gender of referents if needed. as such, the gender of referents in the generated mt output depends entirely on the training data which might contain (un)conscious biases that are transmitted in (written and spoken) datasets. ## related work recent years, several datasets were created that focus specifically on gender-related issues observed in (sub)fields of natural language processing (nlp). targeted gender datasets (test sets or corpora) exist for subfield such as coreference resolution @xcite @xcite and sentiment analysis @xcite . in this section, we will limit our discussion to datasets created specifically for mitigating and assessing gender bias in mt. in the field of mt, mirkin and meunier (2015) used a recommender system approach to predicted user-based preferred translations based on preferences of similar users. @xcite worked on personalized statistical mt. their work centers around the preservation of gender traits by treating gender as a separate domain. for their experiments, they created a bilingual parallel corpus (english-french and english-german) annotated, among others, with the gender of the speaker. 4 for neural mt, @xcite experimented with the integration of speaker gender-tags added to the source side of the parallel corpus. using the demographic information released by @xcite , they compiled large datasets with gender information for 20 language pairs. foot_3 both papers @xcite focused specifically on gender agreement with the first person singular. as such, their corpora are limited to sentence-level gender-tags indicating the gender of the speaker. stanovsky et al. (2019) presented "winomt" a challenge set for the evaluation of gender bias in mt. the set is based on two existing data sets for gender bias in coreference resolution: winobias @xcite and winogender @xcite . winobias and winogender consist of english sentences with two human entities in the form of two gender-neutral occupations (e.g. 'teacher', 'mechanic','assistant'...) and a gendered pronoun referring to one of the two human referents. winomt is a concatenation of winobias and winogender and contains a total of 3,888 synthetic english sentences balanced for gender. the main contribution in @xcite is an automatic evaluation of six popular mt systems on eight language pairs. foot_4 they provide an automatic gender bias evaluation protocol and show that the level of agreement with human annotations is above 85% for all languages. costa-jussà et al. (2020) presents the 'gebiotoolkit', a toolkit for the extraction of gender-balanced multilingual corpora with document-level gender annotations. they also introduce two versions of the 'gebiocorpus'. the first one contains 16k sentences used for evaluating the automatically extracted parallel sentences. from the evaluation, it resulted that the human annotators gave the tool on average a 87.5% accuracy. the second version is a high-quality non-synthetic set of 2k english, spanish and catalan sentences post-edited by native speakers. saunders and byrne (2020) created a small hand-crafted set of gender-balanced sentences for model adaptation. the set consists of 388 english synthetic sentences containing professions and their manually generated translations in each target language (hebrew, german and spanish). @xcite explore the potential of explicit word-level gender inflection tags showing promising results. as such, gender tagging could be an effective tool for automatic translation tools where the user could specify the desired gender of the referents. our english-italian parallel challenge set contains natural sentences (as opposed to synthetic) that do no follow a specific pattern foot_5 with wordlevel gender inflection tags. since naturally occurring sentences are more complex and can contain multiple entities, animate nouns and pronouns have been annotated with word-level tags that indicate the gender given the limited sentence-level context. unlike previous work, the challenge set is not limited to specific phenomena (e.g. 1 st or 3 rd person singular) but covers the full range of natural gender phenomena. it is specifically designed to encourage work on controlling output in terms of gender, the identification of gender ambiguous sentences and co-reference resolution. ## creation and annotation of dataset in this section, we describe the pre-processing, cleaning and the gender annotations steps. ## analysis and discussion this section provides an analysis and discussion of the specific problems posed by the italian language and the specific choices taken with respect 9 in practice and for consistency, we verified whenever a full name was given whether the referent has a wikipedia page on which they are being referred to with specific pronouns. 10 as outlined in the previous section, when there are multiple ambiguous referents we added an additional identifier (1-...) to indicate whether a sentence contains multiple ambiguous entities as this might have an influence on the amount of different gendered translations. 11 annotations were provided by linguists and the italian translations were generated by a native italian speaker/linguist. to the gender translations proposed in the corpus. first of all, italian is a pro-drop language and the subject pronoun is often omitted. therefore in sentences where there are ambiguous subjects (i, you, we, they), like in: example 4.1. "why did @xmath0. however, if there are adjectives referring to ambiguous pronouns, gendered translations are needed, e.g.: example 4.2. "@xmath1, @xmath2." for which we have two italian sentences, namely "disse: "sono stufo di questa cosa chiamata new jersey" for the masculine form and 'disse: "sono stufa di questa cosa chiamata new jersey"' for the feminine form. the same applies when there are past participle forms in the sentence, since in italian these forms sometimes require gender agreement with the noun they refer to such as in: example 4.3. "what did @xmath3: "come pensavi che ti saresti sentito?" and "come pensavi che ti saresti sentita?". gender translations were needed for bigender italian nouns as well, such as for instance insegnante (teacher) or paziente (patient), which have a single invariable form for masculine and feminine and the gender becomes apparent only when there is a coordinated article or adjective , such as in example 4.4. "do @xmath4. we also made a conscious decision in terms of the italian 'non-marked' masculine form, also called the inclusive masculine -when the masculine form is used to refer, generically to males and females, such as for instance the use of the masculine form bambini (children) to refer to both male and female children. for this particular form, although the use of the inclusive masculine is acceptable when refering to a group of people whose gender is unknown (as proposed in e.g. robustelli (2012)), we still opted to provide an alternative translation. for instance the sentence: example 4.5. "man, @xmath5, drawers are open, clothes hanging outside the drawers, the @xmath6..." is translated as: "amico, torno a casa dal lavoro, i cassetti sono aperti, i vestiti tutti fuori, i bambini sono ancora in pigiama..." and "amico, torno a casa dal lavoro, i cassetti sono aperti, i vestiti tutti fuori, le bambine sono ancora in pigiama..." the generic masculine form is also used for the agreement of adjectives/past participles/nouns in agreement with the natural gender of referents that have different genders, e.g.: "giovanni e lucia sono bravi insegnanti" (giovanni and lucia are good teachers). in this case, we kept the masculine form as no possible alternatives are currently accepted. recently, the use of the schwa, "@", was proposed @xcite , precisely to solve these types of problems related to the use of the inclusive masculine form but also to take into account non-binary people representation needs, nevertheless this solution has not yet been widely adopted and is not accepted as a linguistic norm. a further problem addressed in providing gender translation is related to the so-called agentive nouns, namely those nouns that are used to classify people that have specific functions, roles, professions. this type of nouns represent the main problem of sexism in the italian language, and it is currently widely debated, since the tendency is to use male forms also to refer to professions or roles played by women. this is especially true for nouns which refer to particularly prestigious roles, such as direttore (director), presidente (president), ministro (minister), professore (professor) and the like, for which feminine nouns exist: direttrice (female director), presidentessa (female president), ministra (female minister), professoressa (female professor), etc. these forms are not always used (including by women) as some consider the feminization of a profession a loss of prestige. 12 for these cases, we opted to provide both masculine and feminine translations: example 4.6. "so @xmath7." for which we provide the following alternate translations: "e così un giorno decisi di andare a trovare il direttore" and "e così un giorno decisi di andare a trovare la direttrice." ## conclusions and future work in this paper, we present and describe gender-it: an english-italian annotated parallel challenge set. the english source side is annotated with word-level gender tags, while for the italian target side the translations -including correct gender alternatives-are provided. we present a detailed description of the annotations as well as a contrastive analysis of translation specific gender challenges for english-italian. in future work, we envisage working on: (i) an extension of the corpus to other languages, (ii) the identification of gender ambiguous sentences in english, and (iii) the subsequent generation of multiple gender alternatives where necessary, including paraphrases to adopt more gender-neutral solutions. with our challenge set and analysis, we hope to encourage research on ambiguity detection and the controlled generation of gender diverse alternatives for translations.