Gender Bias in E nglish-to- G reek Machine Translation (2025)

Abstract: As the demand for inclusive language increases, concern has grown over the susceptibility of machine translation (MT) systems to reinforce gender stereotypes. This study investigates gender bias in two commercial MT systems, Google Translate and DeepL, focusing on the understudied English-to-Greek language pair. We address three aspects of gender bias: i) male bias, ii) occupational stereotyping, and iii) errors in anti-stereotypical translations. Additionally, we explore the potential of prompted GPT-4o as a bias mitigation tool that provides both gender-explicit and gender-neutral alternatives when necessary. To achieve this, we introduce GendEL, a manually crafted bilingual dataset of 240 gender-ambiguous and unambiguous sentences that feature stereotypical occupational nouns and adjectives. We find persistent gender bias in translations by both MT systems; while they perform well in cases where gender is explicitly defined, with DeepL outperforming both Google Translate and GPT-4o in feminine gender-unambiguous sentences, they are far from producing gender-inclusive or neutral translations when the gender is unspecified. GPT-4o shows promise, generating appropriate gendered and neutral alternatives for most ambiguous cases, though residual biases remain evident. As one of the first comprehensive studies on gender bias in English-to-Greek MT, we provide both our data and code at https://github.com/elenigkove/genderbias_EN-EL_MT .

## introduction as the demand for inclusive language grows, the role of technology in shaping linguistic norms becomes increasingly important. while mt is widely used for communication, cost reduction, and accessibility @xcite @xcite and despite advancements in the field with state-of-the-art neural machine translation (nmt) systems, they often produce inaccurate, ungrammatical, or biased translations, particularly in assigning gender @xcite . concern has therefore grown over the susceptibility of those systems to translate based on gender stereotypes and the perpetuation of such biases via them, as they can have real-word, harmful consequences for users and society @xcite @xcite . a mt model is considered biased "when it systematically and unfairly discriminates against certain individuals or groups in favour of others" @xcite . while human translators rely on the wider context to determine the appropriate gender, most current mt systems do not; instead, they rely on spurious correlations in the (biased) training data which often lead to defaulting to either male or female forms @xcite . these biases reflect the gender stereotypes that are present in our society. as @xcite highlight, translations are more accurate for sentences involving men due to the training data naturally featuring men more than women, as well as for sentences that align with stereotypical gender roles. for example, references of "male doctors" are more reliably translated than those of "male nurses" @xcite , while more errors are detected when the source texts exhibit anti-stereotypical professions, e.g. "female doctors" or "male nurses" @xcite . this study focuses on gender bias that occurs when translating from english, a notional gender language where gender is not always specified, into modern greek (henceforth greek), a grammatical gender language where it is morphologically and semantically necessary to mark the gender @xcite . while prior work has focused on high-resource languages such as german, spanish, and french @xcite @xcite , greek remains understudied despite preliminary exploration on document-level @xcite and labor-domain bias analysis @xcite . these cross-linguistic differences can lead to ambiguities that are difficult to resolve, especially for sentence-level mt systems @xcite , making it more complex to accurately assign gender or maintain gender neutrality. this raises important questions as to what constitutes appropriate handling of gender-ambiguous inputs. although individual translations (e.g., translating "the worker" as either feminine or masculine) may be grammatically valid, aggregate patterns reveal systemic biases. we argue that ideal mt systems should recognise when gender cannot be determined from context and provide either gender-neutral options, or a range of plausible gendered interpretations. this framework informs our evaluation of existing systems and our proposed llm-based mitigation approach. the main contribution of this work includes: ## results a preliminary analysis reveals substantial differences in gender and error distributions across the three models ( regarding errors, google translate had the highest rates: 13.3% of translations included mixed genders, 3.7% were incorrect/nonsensical translations, and 0.4% omitted the adjective. deepl performed better, with only 1.3% mixed-gender and 1.3% incorrect/nonsensical translations. gpt-4o also showed low error rates (1.7% mixed-gender, 1.3% incorrect/nonsensical) but omitted adjectives in 4.2% of cases. interestingly, gpt-4o introduced a unique error type; as the only model that actively attempted to provide neutral forms, it did not always succeed, resulting in 3.3% of cases that contained errors related to the neutralisation techniques. ## discussion our study confirms significant gender bias in english-to-greek mt, with both google translate and deepl defaulting to masculine forms for gender-ambiguous contexts, rendering it consistent with findings in other language pairs @xcite @xcite . while this male bias persisted across occupational stereotypes, increased feminine forms for stereotypically female occupations demonstrate how systems simultaneously reinforce male defaults and societal gender associations embedded in the systems' training data @xcite . both mt systems performed consistently well with explicitly masculine referents, supporting previous work showing better handling of male references and stereotypical roles @xcite @xcite . however, google translate struggled more with anti-stereotypical feminine forms. this disparity is likely influenced by differences in training data or model architecture, but the black-box nature of these systems makes it difficult to determine the exact cause. most critically, neither system produced (adequate) gender-neutral outputs, consistently failing on non-binary cases. prompted gpt-4o demonstrated high performance, successfully generating feminine, neutral and masculine alternatives for most ambiguous cases. while it showed residual bias (better accuracy in masculine forms), the model was able to follow instructions to identify gender-ambiguous sentences and generate inclusive output for them. we thus confirmed llms' potential to handle genderaware translation, supporting emerging research @xcite that also demonstrates that gpt is a promising solution for producing genderneutral outputs when given only a few examples. nevertheless, the translation errors (e.g. nonexistent words, incorrect pronouns, missing adjectives) or incorrect neutralisation solutions (e.g. use of epicene nouns, double forms with binary pronouns) that were produced for a small number of gender-neutral outputs, should be taken into consideration. these issues underscore the tension between technological solutions and linguistic reality; that is, an inherent bias in the training data of the model, but also the challenges in adapting genderneutral practices for greek. language reform in greek has followed a slower pace compared to other languages and currently lacks sufficient linguistic structures for a gender-neutral language or structures that address the visibility of underrepresented groups, such as lgbtqia+ individuals and women. as such, the discussion about linguistic sexism and the development of genderinclusive practices is still open and evolving. ## limitations some limitations should be acknowledged. first, there is a reproducibility problem as the study relies on three closed-source models. as proprietary systems subject to frequent updates, the results of the same query may vary across multiple trials. moreover, gpt-4o requires a paid subscription, 16 which limits its accessibility compared to freely available systems. second, while gendel's controlled sentence structures (based on gender-biased adjectives and occupational nouns) ensure methodological consistency, this design restricts the generalisability of the results to more diverse and natural text. similarly, our gpt-4o prompt was specifically customised for these sentence structures, which raises questions about its applicability to more complex data. gendel, therefore, should be viewed as a foundational resource for evaluating english-to-greek gender bias, and future research could supplement a wider variety of sentence structures, contexts, linguistic phenomena, and manifestations of gender bias, as well as further experimentation with llm prompting strategies. ## conclusion in response to the emerging demand for inclusive language, this study focused on the under-represented english-to-greek language pair. through extensive, fine-grained manual analyses and descriptive statistics, we demonstrated that gender bias is persistent in translations by google translate and deepl, highlighting that, while they perform well in cases where the referent's gender is defined, they are far from recognising and producing gender-neutral language. we also demonstrated that gpt-4o, when prompted, can achieve high accuracy on providing gendered and genderneutral alternatives in cases of ambiguity. by situating our results within the context of prior research, this study makes two important contributions: (1) the creation and public release of gen-del, the first handcrafted dataset for evaluating english-to-greek translations, and (2) empirical evidence emphasising the urgent need for more inclusive translation practices in greek. we hope this work will inspire further research on this language pair and contribute to the development of more inclusive translation technologies.