Fine-grained Gender Control in Machine Translation with Large Language Models (2024)

Abstract: In machine translation, the problem of ambiguously gendered input has been pointed out, where the gender of an entity is not available in the source sentence. To address this ambiguity issue, the task of controlled translation that takes the gender of the ambiguous entity as additional input have been proposed. However, most existing works have only considered a simplified setup of one target gender for input. In this paper, we tackle controlled translation in a more realistic setting of inputs with multiple entities and propose Gender-of-Entity (GoE) prompting method for LLMs. Our proposed method instructs the model with fine-grained entity-level gender information to translate with correct gender inflections. By utilizing four evaluation benchmarks, we investigate the controlled translation capability of LLMs in multiple dimensions and find that LLMs reach state-of-the-art performance in controlled translation. Furthermore, we discover an emergence of gender interference phenomenon when controlling the gender of multiple entities. Finally, we address the limitations of existing gender accuracy evaluation metrics and propose leveraging LLMs as an evaluator for gender inflection in machine translation.

## introduction in machine translation (mt) research, many efforts have been made to improve the gender accuracy of nmt systems, which have shown to exhibit gender bias @xcite . this research includes the task of handling ambiguously gendered entities in text, which arises from differences in gender markings across different languages @xcite . without consideration for these ambiguities, existing mt systems default to masculine translations or use a stereotypically associated gender, reflecting the bias in training data @xcite . to address the gender ambiguity issue, multiple approaches have been proposed, such as gendered translation rewriting @xcite , generating gender-neutral translations @xcite , and controlled translation @xcite . specifically, controlled translation methods take the gender of the ambiguous entity as additional input along with the source text, and generate a translation matching the given gender. however, most previous works have only considered gender control of a single entity for each input, and a gap still remains between this simplified experimental setup and texts found in real-world contexts where multiple entities are often mentioned within the same context. in this work, we investigate the task of controlled translation in a more realistic, fine-grained setting where the given text has multiple entities with different gender assignments. to this end, we employ llms and propose gender-of-entity (goe) prompting for fine-grained gender control in machine translation. our method utilizes the powerful instruction-following and translation capabilties of llms for a more accurate translation aligned with the target gender inflections. llms are explicitly instructed to translate the source text with additional entity-level gender information given in natural language statements. for a comprehensive assessment, we employ four existing benchmarks on gender bias evaluation and investigate the llm's capabilities in various scenarios, ranging from sentences with multiple ambiguously gendered entities to sentences containing both unambiguously gendered and ambiguously gendered entities. from our experiments, we find that the goe prompting on the llms scores up to an average of 95.4% gender accuracy on the must-she dataset, significantly outperforming previous control methods based on fine-tuning. fur-thermore, we identify a problematic phenomenon of gender interference in fine-grained controlled translation, where controlling the gender of one entity adversely affects the gender inflection of other entities. these findings emphasize the necessity of fine-grained assessment of gendered entities in gender bias evaluation. finally, we find that conventional metrics used in gender bias evaluation are based on lexical matching, making it challenging to capture synonyms or paraphrases. we thus propose leveraging llms as a reference-free evaluator that checks the gender inflections and agreements of the translation. experimental results show the validity of our proposed evaluator from the high correlation with human judgements and with the automated metrics as well. to summarize our work, in section 2, we formulate the fine-grained controlled machine translation task and introduce the four benchmark datasets used in our paper. next, we report our controlled translation experiments on the four evaluation settings and share our findings in section 3. in section 4, we investigate using llms as a gender evaluator. in section 5, we share related works and conclude our paper in section 6. ## gender control in machine translation we formalize the controlled translation task of gender attributes in machine translation and introduce four gender control scenarios based on existing evaluation benchmarks. we then introduce our proposed llm-based controlled translation methodology. ## main experiments we experiment on controlled translation of gender with our proposed method on the four evaluation benchmarks and compare them with existing approaches. ## llm as gender evaluators in section 3, our methodology exhibits significant performance based on automated gender accuracy metrics. however, the employed coverage-based metrics are dependent on the annotated gender terms. such dependence poses a challenge in assessing gender terms that do not match the annotations due to the use of synonyms or different grammatical structures. for example, the english term "professor" can be translated into either "profesor/profesora" or "maestro/maestra" in spanish. as shown in tables 2, 3, and 5, at least 20% of the samples for each evaluation benchmark remain unevaluated due to missing coverage from the provided gender annotations. as a result, it is necessary to address these issues for a more complete and accurate assessment of gender accuracy. as gender evaluators (lge). we provide llms with instructions as specified in ## conclusion in this paper, we tackled fine-grained gender control in machine translation. to solve this task, we proposed gender-of-entity prompting method for llms, where we instruct llms to translate with additional entity-level gender information given in natural language statements. results on four evaluation benchmark show promising capabilities of llms as controlled translator of gender, with up to 95% average accuracy on the must-she dataset. we also observe a new phenomena of performance degradation when translating sentences with multiple gendered sentences with different target genders, which we refer to as gender interference. finally, we addressed the limitations of existing automated gender evaluation metrics by proposing llms as gender evaluators (lge). based on experimental results, lge evaluations were shown to have high correlation with human judgements.