Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Models (2024)

Abstract: Machine translation (MT) models are known to suffer from gender bias, especially when translating into languages with extensive gendered morphology. Accordingly, they still fall short in using gender-inclusive language, also representative of non-binary identities. In this paper, we look at gender-inclusive neomorphemes, neologistic elements that avoid binary gender markings as an approach towards fairer MT. In this direction, we explore prompting techniques with large language models (LLMs) to translate from English into Italian using neomorphemes. So far, this area has been under-explored due to its novelty and the lack of publicly available evaluation resources. We fill this gap by releasing NEO-GATE, a resource designed to evaluate gender-inclusive en→it translation with neomorphemes. With NEO-GATE, we assess four LLMs of different families and sizes and different prompt formats, identifying strengths and weaknesses of each on this novel task for MT.

## introduction machine translation (mt) has been found to be susceptible to gender bias, i.e. the tendency to produce default masculine outputs or stereotypical gender associations @xcite @xcite when gender information about human referents is absent. this is especially relevant when translating from notional gender languages like english, which express gender only through a limited set of elements (e.g., he/she pronouns), into grammatical gender target languages, such as italian, german, and spanish, which mark gender extensively in their morphology (e.g., en: "my friends are rich" → it: "i miei amici sono ricchi" [m] vs "le mie amiche sono ricche" @xcite ). the consequences of this behavior are systematically harmful @xcite to women, who risk being under-represented and stereotypically defined, and non-binary individuals, who are erased from representation or misgendered within binary gender linguistic frameworks @xcite . in light of this, in this paper we look at neologistic solutions -which are emerging from grassroots efforts to make language more inclusive -as a path towards gender-inclusive mt. linguistic innovations such as neopronouns (e.g., en ze instead of he/she, sw hen instead of han/hon) and neomorphemes (e.g., it -@/-3, es -e/-es in place of gendered inflectional morphemes) add new elements to the grammar and morphology to allow for the expression of non-binary gender identities or to convey gender neutrality @xcite . to date, the use of neologistic solutions is not systematized yet, with alternative paradigms coexisting and new ones continuously emerging. the choice and use of one paradigm of neologistic devices (e.g., the neopronouns xe/xem/xyr/xyrs/xemself vs ze/zir/zir/zirs/zirself, etc.) depends on individuals' identity and preferences in gender expression. the use of neologistic devices in mt is still a largely unexplored research area, due to the novelty of this approach and to the lack of dedicated en i like being surrounded by my friends. a mi piace essere circondato dai miei amici. b mi piace avere persone amiche intorno a me. to this aim, we i) release neo-gate, a benchmark to evaluate gender-inclusive @xmath1; ii) explore different prompting strategies for neologistic gender-inclusive mt, across three open and one commercial llms, and the two most popular italian neomorpheme paradigms. ## background following evolving social and linguistic phenomena @xcite , there has been a rising demand for the integration of gender-inclusive language in natural language processing (nlp) technologies to make them inclusive of all gender identities @xcite . in mt, gender-neutral translation (gnt) was recently proposed as a gender-fair approach to make translation technologies less biased and more inclusive @xcite . gnt consists in using gender-neutralization strategies, such as epicene formulations, (e.g., 'persone' -en: people -in examples b and c in in this work, we focus on the use of neomorphemes in @xmath3, a scenario in which gender-related ambiguities -and, consequently, the need for gender-inclusive solutions -are crucial. indeed, italian is characterized by a pervasive gender-marking system, which assigns a gender to each noun and every word syntactically linked to it, including some verbal forms. coherently, there have been several proposals of neomorpheme paradigms, which currently coexist and are not yet ultimately codified @xcite . such proposals promote the use of specific characters in place of gendered morphemes (e.g., masculine -o and feminine -a, as in "uno scienziato" [m], "una scienziata" [f] -en: a scientist). the proposed neomorphemes range from letters of the latin alphabet (e.g., 'u'→unu scienziatu), to typographical symbols (e.g., '*'→un* scienziat*), to letters of the international phonetic alphabet (ipa), like the schwa neomorpheme paradigm, which uses the ipa letter '@' for the singular number (un@ scienziat@) and '3' for the plural (alcun3 scienziat3 -en: a few scientists) @xcite . gender inclusivity in nlp so far, research on gender-inclusive neologistic solutions in nlp has been mainly limited to first explorations in monolingual settings, and mostly confined to english neopronouns. in a pioneering effort, @xcite discussed the adoption of neopronouns and formulated a list of desiderata to model the use of pronouns in language technologies. they redefined pronouns as an open class, i.e., a class which is not fixed and allows for the inclusion of emerging neopronoun paradigms each user may identify with. this is crucial when dealing with such novel and constantly evolving devices. in the context of generative tasks, several studies highlight the difficulty of llms in handling neopronouns in zero-shot settings @xcite @xcite . @xcite identify byte pair encoding tokenization @xcite as a major cause of llms' shortcomings, coherently with @xcite , which observed the same phenomenon in gender bias investigation. tokenization, paired with the un-fixed nature of innovative gender-inclusive solutions, may represent a crucial problem for llms in correctly generating neomorphemes as well. indeed, as mentioned above, the range of characters used as neomorphemes is wide and i) not all characters are necessarily represented in the training data of llms; ii) the use of different characters in place of more common gendered morphemes may result in different tokenizations for otherwise identical terms, which in turn could interfere with llms' ability to generate fluent text. in mt research, the sole experiment in developing systems partially compatible with neologistic devices is a proof-of-concept built by @xcite in a gender bias mitigation experiment. they fine-tuned @xmath4, to be replaced with non-binary forms post-inference. in a broader analysis of gender bias in llms, vanmassenhove (2024) reports that chatgpt never produces gender-inclusive neomorphemes when translating ambiguous english sentences into italian, although without specifically prompting the model to do so. the sole analysis dedicated to the use of neologistic devices in mt is the one by @xcite , which shows how commercial systems fail to deal with english neopronouns, resulting in either misgendering or low-quality outputs. a major bottleneck hindering the exploration of gender-inclusive neologistic devices in mt is the lack of publicly available evaluation resources. to bridge this gap, in the next section we introduce a dedicated resource: neo-gate. neo-gate is built upon gate @xcite , a benchmark for the evaluation of gender bias in mt. in gate, the gender of human entities is unknown, i.e. there are no elements providing gender information about human referents in the (english) source sentences. gate also provides target language references which only differ in the feminine/masculine gendered words that refer to human entities (see ## conclusions we discussed a neologistic approach to genderinclusive machine translation, an underexplored area constrained by the lack of publicly available dedicated data. our first contribution, the release of the neo-gate benchmark, allowed us to give a first fundamental impulse to research in this direction. as a second contribution, we explored the possibility of performing gender-inclusive translation from english to italian with four popular large language models: three open models -mixtral, tower, and llama 2 -and a commercial one -gpt-4. our comparisons across different prompting settings reveal that gpt-4 and mixtral generally exhibit promising results when properly prompted, while llama 2 and tower are unfit for the task. more specifically, models' understanding of the task is significantly influenced by prompt complexity, the number of demonstrations, and the specific characters employed as neomorphemes (possibly depending on the representation of those characters in each model's training data). while our investigation suggests llms' potential for neologistic gender-inclusive mt, there remains room for improving their accuracy. neo-gate and the analyses presented herein lay the groundwork for rising to the challenge and for future research on gender-inclusive mt tailored to existing neologistic paradigms, and those that may emerge in this new and evolving landscape.