Gender-specific Machine Translation with Large Language Models (2024)

Abstract: ‘While machine translation (MT) systems have seen significant improvements,it is still common for translations to reflect societal biases, such as genderbias. Decoder-only language models (LLMs) have demonstrated potential in MT, albeitwith performance slightly lagging behind traditional encoder-decoder neural machinetranslation (NMT) systems. However, LLMs offer a unique advantage: the abilityto control the properties of the output through prompting. In this study, we leveragethis flexibility to explore Llama”s capability to produce gender-specific translations.Our results indicate that Llama can generate gender-specific translations withtranslation quality and gender bias comparable to NLLB, a state-of-the-art multilingualNMT system.’

## introduction over the last few years, machine translation (mt) systems have seen significant improvements with the introduction of neural machine translation (nmt). despite these advances, mt can reflect societal biases, such as gender bias. a prominent instance of this problem occurs when the target language marks the grammatical gender, but the source language does not (fig. 1b ). in such instances, translating into either gender can be correct, but mt systems tend to pick the gender that corresponds to stereotypical associations (e.g., associating certain professions to males and others to females @xcite ). instead, it would be preferable to generate both options, and/or let the user control the gender. decoder-only large language models (llms) have shown mt capabilities inferior to but competitive with encoder-decoder neural machine translation (nmt) systems @xcite @xcite @xcite @xcite . however, llms have been translate the following english sentences into spanish. translate the following english sentences into both grammatical genders of spanish. provide two spanish sentences for each english source sentence. figure 1 : prompt templates employed. languages like spanish (displayed here) present morphological differences between masculine (red) and female (purple) grammatical genders. the task of inferring gender-specific translations from a gender-neutral source has been addressed mainly through post-editing, the most popular solution being google translate's post-translation gender rewriter @xcite . the proposed system produces a single sentence that is then switched into the opposite gender using a second languagespecific model. this approach is limited by having to train language-specific gender-switching models and the breadth of patterns it can cover. given the flexibility of prompting, we explore the capacity of llms to produce gender-specific translations for languages with grammatical gender from gender-neutral sources without significant losses in translation quality or increases in gender bias. we use in-context examples (ices) to elicit the task of translation from a gender-neutral source to two gender-specific targets (figure 1b ). additionally, we evaluate the quality of the gender-specific translations on two aspects: gender bias (measured against coreference resolution accuracy) and translation quality (measured in bleu). we show that it is possible to generate genderspecific translations with translation quality and gender bias competitive with nllb, with a slightly better performance than llama for masculine/both references evaluation and over 10 bleu points for the feminine reference. we also demonstrate the reliance on coreference resolution of the genderspecific translation method, showing steep decreases in performance when using the opposite gender as an evaluation reference in a genderfocused dataset (multilingualholisticbias), but exhibiting lesser variance in a general translation dataset (flores). gender bias in mt some authors have worked in analyzing and mitigating gender bias in mt. @xcite studied the bias of the commercial translation system google translate and found that it yields male defaults much more frequently than what would be expected from us demographic data. @xcite investigate the role of model architecture in the level of gender bias, while @xcite looks at the source sentences and elaborates a taxonomy of the features that induce gender bias into the translations. others have looked more closely at the challenge of gender bias mitigation. @xcite assume that it's not always possible to infer all the necessary information from the source sentence alone and a method that uses word-level annotations containing information about the subject's gender to decouple the task of performing an unbiased translation from the task of acquiring gender-specific information. saunders and byrne (2020) treat the mitigation as a domain adaptation problem, using transfer learning on a small set of trusted, gender-balanced examples to achieve considerable gains with a fraction of the from-scratch finally, @xcite focus on existing biases in person name translation, applying a data augmentation technique consisting of randomly switching entities, obtaining satisfactory results. given this work's focus area, we aim not only at producing accurate gender-specific translations, but also at ensuring selecting an output gender does not increase reproduction of underlying gender biases. ## experimental framework data for our main experiments, we use the mul-tilingualholisticbias dataset @xcite , a multilingual subset of holistic bias @xcite with separate translations for each noun class or grammatical gender for those languages that make use of them foot_0 . an example of an entry of the dataset can be found in evaluation following the work of costa-jussà et al. ( 2023 ), we use the sacrebleu implementation of spbleu @xcite to compute the translation quality with 'add-@xmath0. we also provide evaluations in chrf @xcite , comet @xcite al., 2020), bleurt (sellam et al., 2020) and blaser (chen et al., 2023) as alternative metrics. for gender bias evaluation, we use stanovsky et al. (2019)'s reference-less coreference resolution metric. ## conclusions in this paper, we explored the capabilities and limitations a decoder-only llm to produce genderspecific translations. we observed that llama's gender-specific translations' accuracy is consistently above nllb's. we also showed that llama's gender-specific translations' gender bias is comparable to nllb's. these results indicate that it is possible to use llms to produce gender-specific translations without compromising on lower translation accuracy or higher gender bias. our experiments also reveal that llama's translations rely on coreference resolution to determine gender, showing significant performance drops when evaluated against opposite-gender references in genderambiguous datasets, but maintaining consistency in less ambiguous contexts. while these results are promising indicator of the flexibility of the output in the task of mt for languages present in llama's training set, the limited multilinguality of currently available llms limits the application of this approach to a subset of the languages present in state-of-the-art nmt models. more work is needed to bring llms' multilingual capabilities on par with nmts.