Building Bridges: A Dataset for Evaluating Gender-Fair Machine Translation into G erman (2024)

Abstract: The translation of gender-neutral person-referring terms (e.g.,the students) is often non-trivial.Translating from English into German poses an interesting case—in German, person-referring nouns are usually gender-specific, and if the gender of the referent(s) is unknown or diverse, the generic masculine (die Studenten (m.)) is commonly used. This solution, however, reduces the visibility of other genders, such as women and non-binary people. To counteract gender discrimination, a societal movement towards using gender-fair language exists (e.g., by adopting neosystems). However, gender-fair German is currently barely supported in machine translation (MT), requiring post-editing or manual translations. We address this research gap by studying gender-fair language in English-to-German MT. Concretely, we enrich a community-created gender-fair language dictionary and sample multi-sentence test instances from encyclopedic text and parliamentary speeches.Using these novel resources, we conduct the first benchmark study involving two commercial systems and six neural MT models for translating words in isolation and natural contexts across two domains. Our findings show that most systems produce mainly masculine forms and rarely gender-neutral variants, highlighting the need for future research. We release code and data at https://github.com/g8a9/building-bridges-gender-fair-german-mt.

## introduction gender equality is one of the united nation's sustainable development goals. 1 as psychological research shows that linguistic forms influence the mental representation of gender identities @xcite the work cannot be delegated to the praesidium, to the triumvirate of the chairman and his two deputies. ## data for gender-fair mt we release two resources for studying gfl in english-to-german mt. first, we assemble a dictionary ( §3.1) of person-referring nouns. second, we sample passages from wikipedia and europarl ( §3.2) to study our terms in natural contexts. words in isolation allow for testing priors in translation systems, i.e., the most likely gender form for a noun when no context is provided. natural passages enable studying the effect of contextual clues. note that while this work focuses on evaluating german translations, our resource can be enriched with any grammatical gender language where gender-fair language approaches ought to be preferred to masculine generics (e.g., @xcite . ## related work due to stereotypical and exclusive biases present in the training data, the output of mt may discriminate against certain genders (e.g., @xcite . in this context, recent research has focused on the issue of gender exclusivity @xcite . towards a better understanding, much attention has been paid to studying existing strategies chosen by human subjects, like translation team leaders @xcite , and mt post editors @xcite . related to this, gromann et al. (2023) pointed to participatory research as a promising avenue. another research thread focuses on assessing the capabilities of existing mt systems: lauscher et al. (2023) investigated the translation of pronouns in commercial mt, saunders and olsen (2023) the translation of named entities, and piergentili et al. (2023b) benchmarked genderneutral mt from english to italian. savoldi et al. @xcite report the results of a shared task, designed to assess the gfl ability of mt systems from german to english. the only existing work, which also focuses, like ours, on english to german gfl is @xcite . however, the authors study 15 sentences only. in contrast, we focus on 115 words in multiple translation scenarios. ## conclusion we have presented the first study on gender-fair en-de mt. we introduced two novel resources grounded in community contributions. we experimented with eight translation systems and several setups, including words in isolation, natural passages from the encyclopedic domain and parliamentary speeches. our findings call for more research on gfl in modern mt towards fairer and more inclusive translation technology.