GENDEROUS : Machine Translation and Cross-Linguistic Evaluation of a Gender-Ambiguous Dataset (2025)

Abstract: Contributing to research on gender beyond the binary, this work introduces GENDEROUS, a dataset of gender-ambiguous sentences containing gender-marked occupations and adjectives, and sentences with the ambiguous or non-binary pronoun their. We cross-linguistically evaluate how machine translation (MT) systems and large language models (LLMs) translate these sentences from English into four grammatical gender languages: Greek, German, Spanish and Dutch. We show the systems’ continued default to male-gendered translations, with exceptions (particularly for Dutch). Prompting for alternatives, however, shows potential in attaining more diverse and neutral translations across all languages. An LLM-as-a-judge approach was implemented, where benchmarking against gold standards emphasises the continued need for human annotations.

## introduction recent advancements in machine translation (mt) and large language models (llms) have improved translation quality to so-called near-human performance levels @xcite . despite these improvements, systems exhibit gender bias by "systematically and unfairly discriminat @xcite against certain individuals or groups of individuals in favor of others" @xcite . extensive research shows that mt systems and llms continue to struggle with bias and fairness @xcite @xcite , contributing to the discrimination of underrepresented social groups. studies on gender bias in mt have predominantly focussed on higher resource languages and binary gender, with few recent studies focussing on gender neutrality @xcite , less-represented languages @xcite , and non-binarity in translation @xcite @xcite . additionally, while most gender bias challenge sets focus on unambiguous sentences, we need a better understanding of mt translation for gender-ambiguous sentences to mitigate mt gender bias @xcite . to address these research gaps, this paper introduces genderous foot_0 , a dataset of english genderambiguous sentences -constructed without grammatical gender cues, and reports how mt systems and llms inherently translate these into four grammatical gender languages: greek (low resource), german (high resource), spanish (high resource) and dutch (medium resource). we analyse how the translations differ in terms of gender for the four target languages and evaluate the extent to which these systems and languages continue to default to male translations @xcite . we further investigate whether the stereotypicality of an occupational noun, the presence of a gender-inflected adjective, or the interplay between both influence the gender assignment in the translations @xcite as well as what the impact of the presence of the pronoun their is [rq3] -which could be considered ambiguous or as a reference to non-binary individuals. we further explore the impact of prompt-based interventions designed to elicit gender-alternative translations from llms. this allows us to assess whether prompting increases gender diversity or neutrality in output @xcite . given the timeconsuming nature of human gender annotations, we additionally explore the potential of llms for automatic gender annotation by comparing llm labels with the human-annotated gold standard @xcite . prior work has mainly evaluated occupational gender bias in translation using coreference test suites such as winomt @xcite and winobias @xcite -in which each sentence contains a primary entity, referred to with an occupational noun, which is co-referent with a pronoun foot_1 . troles and schmid (2021) extended this dataset by combining occupations with gender-stereotypical adjectives foot_2 and verbs foot_3 . these word types usually further compound occupationspecific bias, as they also carry gender bias in their word-embeddings @xcite @xcite . our work focusses exclusively on the interplay between stereotypical occupations and adjectives in translation. unlike prior studies, we deliberately avoid pronoun co-reference to examine how certain (biased) lexical items shape gender assignments in exclusively gender-ambiguous cases. language comparison studies on gender bias in mt and llms have primarily examined translations from english into high-resource grammatical gender languages like german, french, and spanish @xcite @xcite @xcite @xcite . lower-resource languages such as arabic @xcite , ukrainian @xcite , or polish @xcite -among others -have also been included in evaluation studies; however, two languages have received minimal attention: dutch, evolving from a language with three grammatical genders (masculine/feminine/neuter) to a common/neuter gender system with emerging gender-neutral pronouns @xcite , and greek, whose deeply embedded grammatical gender presents particular challenges. research shows that dutch word embeddings contain gender bias (mulsa and spanakis, 2020) and that bias is present in llm output for story generation @xcite . to the best of our knowledge, this is the first study to explore mt gender bias for dutch. greek gender bias research saw a preliminary exploration in karastergiou and diamantopoulos (2024)'s analysis of documentlevel outputs, followed by mastromichalakis et al. (2024)'s labor-domain bias analysis via knowledge graphs, and gkovedarou et al. (2025)'s comprehensive sentence-level study, which introduced a controlled evaluation of occupational terms and adjective interactions across google translate, deepl, and gpt-4o. our work provides a systematic comparison across both well-documented languages (german, de & spanish, es) and understudied ones (dutch, nl & greek, el). llms & prompting strategies due to their remarkable performance across a variety of natural language processing (nlp) tasks, llms have been tested for their translation capabilities. research shows that they can perform on par with or better than some state-of-the-art mt models, mainly due to the fact that their outputs can be controlled through explicit zero-or few-shot prompt-ing @xcite @xcite @xcite . at the same time, these systems often reinforce gender stereotypes due to inherent biases in their training data @xcite @xcite . work on prompting strategies for gender show that llms do not reliably produce multiple or correct gender alternatives @xcite and that llms struggle to correctly translate the genderambiguous english their from and into lowerresource languages @xcite . our work evaluates both minimal (relying on llms' default behaviour) and controlled prompts (directing gender output) to assess the potential for the reduction of bias. llm as the annotator for the evaluation of the translations, we refrain from using automatic evaluation metrics such as bleu @xcite or ter @xcite , as they fail to adequately capture certain linguistic phenomena, such as gender bias @xcite . instead, we rely on manual evaluation to ensure accurate assessment of the outputs. scaling this can be costly, though; thus, we also explore the llm-as-a-judge paradigm, which has shown promising agreement with human judgments @xcite and has recently been successfully implemented to evaluate gender neutral translations @xcite . as these systems may inherit and amplify the very biases they are meant to evaluate, including both gender stereotypes and methodological biases like positional preference (favoring the first option in pairwise comparisons) @xcite , we compare llm annotations with goldstandard human annotations and calculate interannotator agreement (iaa) to determine the reliability of llms as annotators. ## discussion & conclusion this study offers novel insights into gender bias in mt systems and llms through: (1) the introduction of genderous, a handcrafted dataset of gender-ambiguous sentences, (2) the comparison of two lesser researched languages (nl, el) with more widely investigated ones (de, es), and (3) the analysis of non-binary linguistic forms. rq1: differences in gender distribution across systems & languages our results confirm the persistence of masculine defaults @xcite across both mt systems and llms for genderambiguous occupational terms and reveal how deeply embedded societal stereotypes remain in language technologies, even for artificiallyintelligent models like gpt-4o. across systems, eurollm produced the most errors. our findings confirm patterns for german and spanish established in earlier work, with systems being slightly more inclusive for german than for spanish @xcite . mt systems produced somewhat more female translations than llms, except for greek, where the distribution was similar across systems. dutch was shown to be an outlier. @xcite , where gender in mt translation was strongly influenced by adjective stereotypicality in coreference scenarios, our findings showed that stereotypical male and neutral occupational nouns were predominantly translated as male, regardless of the types of adjectives in the sentence. this indicates that noun stereotypicality appears to be a stronger gender predictor than adjective stereotypicality in ambiguous gender sentences, supporting the need for more research into mt bias in gender-ambiguous scenarios @xcite . merely stereotpyical female nouns were more likely to be translated as female ('assistant' and 'hairdresser' to a lesser extent), particularly in combination with female-inflected adjectives.