Glitter: A Multi-Sentence, Multi-Reference Benchmark for Gender-Fair G erman Machine Translation (2025)

Abstract: Machine translation (MT) research addressing gender inclusivity has gained attention for promoting non-exclusionary language representing all genders. However, existing resources are limited in size, most often consisting of single sentences, or single gender-fair formulation types, leaving questions about MT models’ ability to use context and diverse inclusive forms. We introduce Glitter, an English-German benchmark featuring extended passages with professional translations implementing three gender-fair alternatives: neutral rewording, typographical solutions (gender star), and neologistic forms (-ens forms). Our experiments reveal significant limitations in state-of-the-art language models, which default to masculine generics, struggle to interpret explicit gender cues in context, and rarely produce gender-fair translations. Through a systematic prompting analysis designed to elicit fair language, we demonstrate that these limitations stem from models’ fundamental misunderstanding of gender phenomena, as they fail to implement inclusive forms even when explicitly instructed. Glitter establishes a challenging benchmark, advancing research in gender-fair English-German MT. It highlights substantial room for improvement among leading models and can guide the development of future MT models capable of accurately representing gender diversity.

## introduction language and society are deeply interrelated @xcite , with language both reflecting and reinforcing societal norms and power imbalances. this relationship is particularly evident * the first two authors contributed equally. ## gender-fair language in german grammatical gender languages such as german require gender marking in several word classes, e.g., nouns, pronouns, articles, and adjectives. in the case of person-referring terms, there is an overlap between the term's grammatical gender and the extra-linguistic referent's gender @xcite . this may hinder the linguistic visibility and inclusion of gender identities beyond the binary because, for instance, most european languages have two or three grammatical genders only, i.e., masculine, feminine, and neuter. 2 moreover, said languages rely on masculine forms as generics, reducing the visibility of women and other genders in linguistic and cognitive representation @xcite . this masculine-default bias perpetuates inequality, while gender-fair alternatives promote more inclusive representations @xcite . german has developed several approaches to gender fairness: neutralization replaces gendered terms with neutral alternatives ("die studierenden" instead of "die studenten"); coordination explicitly includes masculine and feminine forms ("studentinnen und studenten"); typographical solutions incorporate inclusive characters like the gendersternchen ("student*innen"), gender-doppelpunkt ("student:innen"), or gendergap ("student_innen"); gender-fair neosystems, e.g., the ens-forms ("studens"), introduce a new set of morphemes inclusive of all genders @xcite . these strategies vary in acceptance, reflecting ongoing evolution toward greater inclusivity. ## constructing glitter we create glitter through a three-step process (figure 1 ): i) extracting and augmenting english sources to ensure comprehensive coverage of gender phenomena, ii) annotating gender contexts for nuanced evaluation, and iii) developing professionally post-edited references in three distinct gender-fair forms for each source passage. the complete data statement @xcite is given in appendix a. ## benchmarking translation systems we conduct an empirical analysis to assess several mt models' performance on glitter. in particular, we are interested in assessing whether contemporary translation systems are robust to the various setups we propose-ambiguous seeds that require gender-fair forms, unambiguous scenarios with different genders of the referent, location of disambiguation gender cues, and synthetic vs. naturalistic contexts. ## eliciting gender-fair translation we showed that mt systems consistently default to masculine forms when translating genderambiguous english multi-sentence text into german. here, we explore whether lms can be prompted to produce gender-fair translations that appropriately respond to contextual cues. ## conclusion we introduced glitter, a dataset comprising 909 sources and three parallel, professionally curated german translations, totaling 1,785 translation pairs-the largest benchmark for testing genderfair english-german mt to date. glitter brings several innovations including i) longer passages and annotations to study the impact of contextual information, ii) human-written references with three gender-fair reformulations (neutral, genderstar, -ens form) , and a iii) fine-grained span-level annotation layer for analyzing gender-related phenomena. to complement the resource, we presented a language model-based automatic metric and baseline results testing several translation systems and prompting strategies. all our results underscore the same compelling finding: contemporary translation systems still struggle with genderfair language, fail in understanding when and how to use it (even if prompted with extensive guidelines on how to do so or if provided with explicit gender cues in the extensive source context), and generally default to (generic) masculine forms.