Implementing Gender-Inclusivity in MT Output using Automatic Post-Editing with LLM s (2024)

Abstract: This paper investigates the effectiveness of combining machine translation (MT) systems and large language models (LLMs) to produce gender-inclusive translations from English to Spanish. The study uses a multi-step approach where a translation is first generated by an MT engine and then reviewed by an LLM. The results suggest that while LLMs, particularly GPT-4, are successful in generating gender-inclusive post-edited translations and show potential in enhancing fluency, they often introduce unnecessary changes and inconsistencies. The findings underscore the continued necessity for human review in the translation process, highlighting the current limitations of AI systems in handling nuanced tasks like gender-inclusive translation. Also, the study highlights that while the combined approach can improve translation fluency, the effectiveness and reliability of the post-edited translations can vary based on the language of the prompts used.

## introduction this paper aims to explore whether llms can be effectively utilized for generating genderinclusive translations. the goal is to determine if this technology can handle the task, or if the expertise of a linguist is still necessary, and to what extent. the challenge lies in the fact that neural machine translation engines frequently fall short in producing gender-inclusive output. when style guides mandate gender-inclusivity in the final translation, post-editors have to make extensive modifications, therefore the mt output is not beneficial for them. we are investigating a multi-step approach to machine translation in which the translation is first produced by an mt engine and is then reviewed by an llm, to make it gender-inclusive. the goal is for the llms to streamline this process and reduce the need for extensive human intervention. ## the challenges of inclusive writing gender-inclusive writing involves using language that does not reinforce traditional gender stereotypes or exclude individuals based on their gender identity. it aims to promote equality and respect for all genders by adopting inclusive terminology and avoiding gendered language whenever possible. nowadays, gender-inclusive writing is particularly important as societies worldwide strive for greater gender equality and recognition of diverse gender identities. in this paper, we decided to focus on the translation from english into spanish for several reasons. firstly, this language pair poses several gender bias challenges (as we will see later in the paper). secondly, it is one of the most relevant language pairs from a business perspective for our company. thirdly, we have highly-trusted internal linguists who are native spanish speakers and have experience in the translation and post-editing field. lastly, we have gender-inclusive language style guides available for this language pair, that we used as a starting point for outlining automatic post-editing guidelines. still, this work is part of an ongoing effort to include additional languages in this experiment. as mentioned above, gender-inclusive language presents challenges when translating from english into spanish. this is mainly due to the grammatical structure and inherent gender marking in the spanish language. unlike english, where gender-neutral language is more common, spanish is a grammatical gender language @xcite , that assigns gender to nouns, adjectives, and pronouns. this gender marking extends to articles ('el' for masculine, 'la' for feminine), and even verb conjugations. for example, the english sentence "the doctor saw the patient" can be translated as "el doctor vio al paciente" (masculine doctor, masculine patient), "el doctor vio a la paciente" (masculine doctor, feminine patient), "la doctora vio al paciente" (feminine doctor, masculine patient) or "la doctora vio a la paciente" (feminine doctor, feminine patient). this inherent gender marking in spanish makes it challenging to maintain gender neutrality in translations, especially when dealing with professions, titles, and pronouns. additionally, spanish has fewer gender-neutral alternatives compared to english, which further complicates the task of creating inclusive translations. translators are often requested to navigate these linguistic differences while striving to preserve the intended meaning and promote gender inclusivity in the target language. ## limitations the analysis predominantly relies on the outcomes generated by three ai systems, leaving out a comprehensive perspective of the broad array of machine translation systems and large language models available. the study's focus on a single content type potentially overlooks variations in language use across diverse contents. by examining a limited subset of segments, the study may risk forming a skewed understanding of ai capabilities. solely focusing on one ## conclusions in conclusion, this study has presented a comprehensive analysis of the performance of llms in producing gender-inclusive translations starting from deepl's raw output. the findings indicate that despite certain potential shown by gpt-4 and palm2, the frequent introduction of unnecessary changes, additions, as well as inconsistencies in terminology and tone of voice, raises concerns about their reliability. furthermore, gpt-4 was found to be more successful than palm2 in identifying and fixing gender-bias issues, especially when using an english prompt. this is probably due to the different size of their respective training datasets: gpt-4 was trained on a significantly larger dataset than palm2, which means that gpt-4 has "more knowledge" than palm2. the study also highlighted the potential impact of the prompt's language on the quality of the translation. the necessity for human review remains paramount, as not all gender bias issues were detected and rectified by the systems analyzed. besides, while the use of llms to address gender bias issues in translation effectively mitigates the necessity for substantial human intervention in this particular area, it introduces other complications. specifically, llms can create unnecessary alterations in the post-edited translation, such as inconsistencies in terminology and tone of voice. this, in turn, requires further post-editing effort to correct these unintended changes. therefore, despite the advantages of using llms for reducing gender bias, we can't conclusively state that they decrease the overall workload for the post-editor. further research should delve into the optimization of these systems and their prompts to enhance the accuracy and inclusivity of machine translations.