MT - G en E val: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation (2022)

Abstract: As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased. In particular, gender accuracy in translation can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MT-GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license.

## introduction although neural machine translation (nmt) has made great strides in quality @xcite , evaluations on generic test sets may not tell the whole story. indeed, nmt models are known to make systematic errors in areas like robustness to input perturbations @xcite , disambiguating pronouns in context @xcite , and translating unambiguous human gender @xcite . in particular, genderrelated issues in translation can lead to translations that are inaccurate, ungrammatical, or biased. accordingly, there has been increasing interest in improving machine translation for gendered entities @xcite @xcite @xcite . adequate evaluation benchmarks play an important role in supporting this line of research and improving understanding of how models perform on the task of gender translation accuracy. existing benchmarks have limited diversity in terms of gender phenomena (e.g., focusing on professions), sentence structure (e.g., using templates to construct sentences), or language coverage (see section 4 for more information), making it difficult to gauge how systems perform in terms of both gender and quality simultaneously. to this end, this paper releases a new machine translation gender evaluation benchmark: mt-geneval. we include development data (for model improvements) as well as test data and corresponding metrics (for comprehensive evaluation). mt-geneval is realistic and diverse, includes a wide variety of contexts for gender disambiguation, and is fully balanced by including human-created gender counterfactuals. in its first release, the mt-geneval benchmark covers translation from english into eight target languages, for two genders. mt-geneval focuses on the task of gender accuracy in translation. we define gender accuracy in translation as the extent to which a machine translation output accurately reflects the gender of the humans mentioned in the input, restricted to cases where the gender is explicitly and linguistically disambiguated in the context of the input. thus, in our benchmark we do not consider the grammatical gender on inanimate objects, or cases where the input gender is ambiguous within the given level of context. 2 nach einigem hin und her akzeptierte blacket 50 pfund als vollen ausgleich für die ihm zustehenden gebühren. 2 having served his apprenticeship crookall became a master painter trading at duke street, douglas. tras servir como aprendiz, crookall se convirtió en maestro pintor en la calle duke de douglas. 3 serrano agreed to the restaurant contract as long as he could have a tapas restaurant. serrano traveled to spain @xcite serrano s'est rendu en espagne [. . . ] ## mt-geneval: gender translation accuracy in 8 language pairs for the initial release, mt-geneval covers translations in two genders (female and male) foot_2 from english (en) into eight diverse and widely-spoken target languages: arabic (ar), french (fr), german (de), hindi (hi), italian (it), portuguese (pt), russian (ru), and spanish (es). the source language has limited morphological gender, with gender expressed only on some pronouns and nouns. by contrast, the target languages have extensive grammatical gender and may express gender through morphological markings on a variety of parts of speech including verbs and adjectives, as well as on inanimate objects. in the target languages, human gender often, but not always, lines up with grammatical gender. in order to facilitate evaluation and training of gender-accurate machine translation systems, we release two test subsets (counterfactual, ## gender evaluation benchmarks for mt in this section, we review existing benchmarks on gender accuracy in machine translation, in order contextualize mt-geneval with respect to similar work. winomt one of the most widely-used datasets for evaluating gender accuracy in machine translation is winomt @xcite , which was created by combining the winogender @xcite and wino-bias @xcite coreference test sets. as such, winomt contains synthetic winograd-style sentences where gender is associated with prostereotypical and anti-stereotypical professions. 10 while the dataset does not contain reference translations, it instead includes a target language-specific alignment-based automatic metric. winomt is one of the gender translation accuracy benchmarks with the largest language coverage: the metric covers translation from english into ar, czech (cs), de, es, fr, hebrew (he), it, polish (pl), ru, and ukrainian (uk). must-she must-she @xcite is a dataset of roughly 1,000 gender-specific segments for each of @xmath0, fr, and it. segments include both text and audio, and are extracted from ted talks @xcite . the dataset contains segments where gender is disambiguated by the intra-sentential context, as well as segments where gender is only present as speaker metadata. it is curated to only include segments containing at least one gender-neutral source word that requires gender marking in the translation. the dataset also provides contrastive references for each segment. must-she was also extended with annotations of agreement chain and part of speech by @xcite , and with source-side gender annotations by @xcite . 5 benchmarking models with mt-geneval in this section, we benchmark both commercial and research-scale models using mt-geneval. in addition to giving initial baseline numbers for mt-geneval, through these experiments we aim to show that mt-geneval is a useful new benchmark. specifically, we show that: ## conclusions in this paper, we have introduced mt-geneval, a counterfactual and contextual benchmark for evaluating gender accuracy in translation from en into ar, de, es, fr, hi, it, pt, and ru. in addition to the test data and evaluation metrics, we are releasing 2400 segments of development data. 18 we have shown that this benchmark is useful for evaluating both commercial and research systems, including contextual machine translation models and genderbalanced models, in terms of gender accuracy as well as quality. we hope that this benchmark and development data will spur more research in the field of gender accuracy in translation on diverse languages.