A Prompt Response to the Demand for Automatic Gender-Neutral Translation (2024)

Abstract: Gender-neutral translation (GNT) that avoids biased and undue binary assumptions is a pivotal challenge for the creation of more inclusive translation technologies. Advancements for this task in Machine Translation (MT), however, are hindered by the lack of dedicated parallel data, which are necessary to adapt MT systems to satisfy neutral constraints. For such a scenario, large language models offer hitherto unforeseen possibilities, as they come with the distinct advantage of being versatile in various (sub)tasks when provided with explicit instructions. In this paper, we explore this potential to automate GNT by comparing MT with the popular GPT-4 model. Through extensive manual analyses, our study empirically reveals the inherent limitations of current MT systems in generating GNTs and provides valuable insights into the potential and challenges associated with prompting for neutrality.

## introduction to foster greater inclusivity in our communication practices, there has been a rise in the adoption of gender-neutral language strategies @xcite , which challenge gender norms and embrace all identities by eschewing unnecessary gendered terms (e.g. police officer vs policeman). such strategies are now widespread across various domains -including institutions @xcite , academia (apa, 2020), and industry @xcite , with their consequential investigation for various natural language processing (nlp) technologies (cao and @xcite @xcite . while recent advancements in nlp have seen the modeling of neutral language into monolingual applications @xcite @xcite , research in cross-lingual settings is relatively limited. previous works in mt (costa-jussà and de jorge, 2020; @xcite @xcite have been mostly confined within binary perspectives to improve the generation of masculine/feminine forms into grammatical gender languages (e.g. @xmath0: dottori/esse). 1 under realistic scenarios though, systems often encounter ambiguous input sentences that do not convey gender distinctions @xcite , and for which gnt would be preferable to prevent undue gender assignments in the target language (e.g. en: @xmath1: personale medico[the medical staff]). despite individual studies indicating that existing mt systems are ill-equipped to handle neutrality @xcite @xcite , the automation of gnt remains an open challenge, hampered by the lack of dedicated resources. to the best of our knowledge, the work by @xcite stands as the sole effort to create gender-neutral mt models, but their fine-tuning approach does not generalize from their small artificial adaptation set. within this landscape, large language models (llms) can offer a solution to meet the demand for gender neutrality, thanks to their adaptability to perform new (sub)tasks based on explicit instructions and few examples @xcite . in fact, albeit llms still lag slightly behind traditional mt in overall translation quality @xcite @xcite , their versatility for controlling specific aspects in the output translation was proven for several attributes @xcite @xcite . in this paper, we thus seek to advance the automation of neutral translation by exploring the po-tential of instruction-following models. to this aim, we focus on @xmath2. by experimenting with different prompts and shot-exemplars, we conduct a fine-grained, manual evaluation showing that: i) used as is neither mt nor gpt are suitable for gnt, but prompting gpt shows surprising neutralization capabilities elicited with just a few examples; ii) while including test set terms as neutralization exemplars in the prompts leads to slightly better gnt performance, gpt can generalize well also when provided with unseen examples. finally, extensive manual evaluations unveil that iii) judging the quality and acceptability of automatic gnt is a subjective task, with notable variations across annotators. to promote future research, we make all our manual output annotations freely available at: https://mt.fbk.eu/gente/ . foot_1 2 methods and settings test set. we run our experiments on gente @xcite , a recently released parallel test set designed to evaluate models' gnt capabilitites. built on europarl data @xcite , it allows us to test mt on naturalistic instances for en-it, a language pair that is highly representative of the challenges of performing gnt into languages with extensive gendered morphology. for such languages, neutral strategies can range from simple word changes (e.g. omissions or synonyms) to complex reformulations that can alter the sentence structure @xcite . hence, generating suitable gnts is a delicate and difficult task, to be carefully weighted not to impact the acceptability of a translation. here, we use a portion of gente consisting of 750 english sentences that are gender-ambiguous, foot_2 and which are thus to be neutrally translated so as to avoid any undue gender inference in italian (e.g. i, with all my colleagues wish to..., it-m: io, con tutti i colleghi desidero... → it-gnt: io, con ogni collega[each colleague], desidero...). 4 systems. as mt models, we select two stateof-the-art commercial systems: amazon translate foot_4 and deepl. 6 for gnt-prompting, we use gpt (gpt-4-0613), which achieved promising results in translation @xcite , though especially for high-resource languages @xcite . as an instructionfollowing model @xcite , gpt is suited to keep adherence to provided guidance when performing a task, a valuable aspect to control the neutral translation of gendered terms. experiments. we explore models' neutralization abilities under two experimental settings: i) base-line, to compare if the mt models and gpt in zero-shot conditions foot_6 can perform gnt, without being explicitly instructed/adapted for the task; and ii) gnt-prompting, to leverage gpt potential when prompted with dedicated instructions and examples. in both settings, for gpt we use temperature 0.0, since @xcite attested a progressive translation degradation with higher temperature values. before delving into their gnt capabilities, in ## gnt-prompting to elicit gpt's flexibility for neutral translations, in the gnt-prompting condition we experiment with three few-shot templates inspired by existing literature on prompting @xcite . our prompts, shown in cot-src q: translate the following english sentence into italian using a gender-neutral language to refer to human entities: [secondly, how far does it increase transparency and accountability of the writers]. think step by step. a: in the english sentence there is one expression which refers to human entities and could be translated in a non-neutral way: <of the writers>. a gender-neutral translation @xmath3. the final gender-neutral translation is [secondariamente, fino a che punto aumenta la trasparenza e la responsabilità di chi scrive?] cot-tgt q: translate the following english sentence into italian using a gender-neutral language to refer to human entities: [secondly, how far does it increase transparency and accountability of the writers?]. think step by step. a: the english sentence can be translated as [secondariamente, fino a che punto aumenta la trasparenza e la responsabilità degli scrittori?]. there is one «expression @xmath4: «<degli @xmath5. a gender-neutral alternative to «<degli @xmath6. the final gender-neutral translation is [secondariamente, fino a che punto aumenta la trasparenza e la responsabilità di chi scrive?]. each prompt is used with 3 exemplar sentences taken from the institutional domain, a context where neutral language is increasingly employed, and which is also covered by gente. to verify gpt's ability to generalize from the provided examples, we experiment with two sets of sentences, which only differ for the inclusion of terms to be neutralized that are either i) present in gentehence seen -or ii) terms that never occur in the test set -hence not seen. we list such terms in ## manual evaluation results in this section, we present the results obtained by all our models in baseline conditions, and by gpt in gnt-prompting conditions. although the assesment of gnt capabilites can be automated with the official gente evaluation protocol, the approach would present two inherent limitations. since the protocol simply classifies whether the examples neut. acc. a src i am pleased to make my contribution. g -out sono lieto di potere contribuire. b src respect for standards lies with the judges. n acc out ... spetta all'autorità giudiziaria. [judicial authority] c src may i quote three actors in this field. n un out posso citare tre persone [people]... d src commissioner, i would like to congratulate the rapporteur. p s-acc out commissario, vorrei congratularmi con chi ha redatto la relazione. [who wrote the report] for each model and prompt, we analyze the same 200 randomly selected and anonymized output sentences, equally distributed across three evaluators -all italian native speakers, highly familiar with a m a z o n d e e p l g p t -4 g p n (a) baseline neutrality. ## conclusions in response to the rising demand for inclusive language (technologies), this study has focused on the possibilities of automating the generation of gender-neutral translations. in particular, given the limitations of general-purpose mt models due to the need for dedicated parallel data, we have explored the potential of gpt to produce genderneutral outputs when translating from english into italian. through extensive, fine-grained manual analyses, we demonstrated that gpt offers promising avenues, as it can grapple with this complex task when given only a few examples and still generalizes beyond them. importantly, our evaluations also show that determining the acceptability of what constitutes a good, acceptable neutral translation comes with notable subjectivity. to enable future research, all our manual output annotations are made available 16 to the community to explore the modeling and assessment of such variability.