M i TT en S : A Dataset for Evaluating Gender Mistranslation (2024)

Abstract: Translation systems, including foundation models capable of translation, can produce errors that result in gender mistranslation, and such errors can be especially harmful. To measure the extent of such potential harms when translating into and out of English, we introduce a dataset, MiTTenS, covering 26 languages from a variety of language families and scripts, including several traditionally under-represented in digital resources. The dataset is constructed with handcrafted passages that target known failure patterns, longer synthetically generated passages, and natural passages sourced from multiple domains. We demonstrate the usefulness of the dataset by evaluating both neural machine translation systems and foundation models, and show that all systems exhibit gender mistranslation and potential harm, even in high resource languages.

## introduction it is well documented that dedicated machine translation systems show forms of gender bias (see @xcite . prior work has highlighted bias when translating from source passages where the meaning is fundamentally ambiguous, in both academic and commercial systems @xcite @xcite . forms of bias have been demonstrated with carefully constructed unambiguous english passages @xcite , and with linguistic constructions targeting specific language pairs @xcite @xcite @xcite . recent advances have enabled general-purpose foundation models with powerful multilingual capabilities including translation @xcite @xcite . these models can be used as building blocks in a wide range of products and applications, highlighting the importance of other work on gender bias in natural language processing more broadly @xcite @xcite . evaluating foundation models raises new challenges of measurement validity, given the wide range of use and potential harms @xcite . skew in training data and measures of bias in underlying models may not be reliable predictors or measurements of potential harm in downstream usage @xcite @xcite . there also remain challenges in empirically measuring performance as systems rapidly improve @xcite , ensuring high quality of service as multilingual capabilities expand @xcite and measuring unintentional harms in new system designs @xcite . in this work, we focus on measuring gender mistranslation in both dedicated translation systems and foundation models that can perform translation. figure 1 illustrates gender mistranslation, and examples of translations that refer to a person in a way that does not reflect the gender identity encoded in the source passage. we focus specifically on gender mistranslation over other harms @xcite , and on expanding coverage of language families and scripts at different levels of digital representation @xcite . adapting evaluation methods to measure gender mistranslation for foundation models presents a few challenges. first, language models are often trained on public internet datasets @xcite which can cause contamination and render evaluation sets mined from public data sources ineffective @xcite . second, gender is encoded in different ways across languages, making it challenging to scale automated evaluation methods. automated methods enable faster modeling iteration, but methods commonly used in translation evaluations (eg, bleu, bleurt) may fail to capture specific dimensions of harm from gender mistranslation. finally, the evolving and contested nature of sociocultural norms related to gender make general purpose benchmark methods challenging to develop, particularly for expressions of non-binary gender across linguistic and cultural contexts globally @xcite @xcite @xcite . to address these challenges, we introduce gender mistranslations test set (mittens); a new dataset with 13 evaluation sets, including 26 languages ( we note that some languages we target such as lingala have few existing evaluation resources. the evaluation sets we release can be expanded in future work (e.g., increasing diversity of source passages, more counterfactual variations). we also leave important challenges with mistranslation of non-binary gender expressions to future work. ## dataset in order to precisely target different constructions and languages, and to enable fine-grained disaggregated evaluation, mittens contains multiple evaluation sets ( ## conclusion we release mittens, a dataset for measuring gender mistranslation harms with 13 evaluation sets that covers 26 languages. this dataset makes progress towards more precisely measuring potential harms and scaling evaluation to more languages. we address challenges with contamination and scoring methods amidst evolving sociocultural norms. future research should measure gender mistranslation in direct translation, expand automated evaluation methods, and to investigate how increasingly capable foundation models might enable interactive or multiple alternative translations. more work is also needed to develop language technologies that produce accurate and faithful representations of non-binary people across all languages.