An LLM -as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation (2025)

Abstract: Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is particularly challenging, with current solutions being limited to monolingual classifiers. Such solutions are not ideal because they do not factor in the source sentence and require dedicated data and fine-tuning to scale to new languages. In this work, we address such limitations by investigating the use of large language models (LLMs) as evaluators of GNT. Specifically, we explore two prompting approaches: one in which LLMs generate sentence-level assessments only, and another—akin to a chain-of-thought approach—where they first produce detailed phrase-level annotations before a sentence-level judgment. Through extensive experiments on multiple languages with five models, both open and proprietary, we show that LLMs can serve as evaluators of GNT. Moreover, we find that prompting for phrase-level annotations before sentence-level assessments consistently improves the accuracy of all models, providing a better and more scalable alternative to current solutions.

## introduction gender-neutral translation (gnt) is the task of translating from one language into another while avoiding gender-specific references in the target text when the source does not provide explicit gender information @xcite . consider the examples in one of the challenges implied by gnt is how to evaluate it automatically, thereby enabling fast, cheap, and replicable assessments. indeed, gnt is a complex open natural language generation task where individual word choices make the difference between success and failure (e.g. en: 'as a teacher' → es: 'como una docente @xcite ) with valid gender-neutralization strategies ranging from pinpointed lexical interventions (example n 1 ) to complex and verbose reformulations (n 2 ) @xcite . the variability in valid neutralization solutions makes gnt a complicated task to evaluate. both traditional and modern mt evaluation metrics struggle to account for these variations, as they are hard to capture at a surface-level @xcite , and neutral outputs are systematically penalized by neural metrics @xcite . currently, the only viable approaches to automatically identify gendered or neutral text rely on dedicated classifiers @xcite , which however do not factor in the source sentence. moreover, so far such solutions were only developed for the evaluation of italian texts, and require dedicated data and training to scale across languages. the lack of easily scalable evaluation solutions hinders advancements in gnt research and system development. to fill this gap, we look at the llm-as-a-judge paradigm @xcite , where large language models (llms) are prompted to perform taskspecific evaluations. specifically, we ask: rq1) can we use llms to evaluate gnt? rq2) does performing intermediate analytical steps improve the accuracy of llm-based gnt evaluations? to answer these questions, we conduct experiments with five llms, evaluating their ability to assess neutrality in italian, spanish, and german texts. we experiment with four evaluation approaches. two approaches focus on generating sentence-level assessments, while the other two simulate a chain-of-thought @xcite , prompting llms to perform fine-grained phraselevel analysis before providing higher-level judgments. overall, we find that llms can serve as evaluators of gnt and that generating phrase-level annotations significantly improves llms' accuracy. ## gnt evaluation prompts to explore the llm-as-a-judge approach to gnt evaluation and investigate whether prompting for intermediate analytical steps leads to higher evaluation accuracy, we experiment with four prompts targeting different approaches to evaluation. we design prompts dedicated to the evaluation of the target language text only (mono) and the source sentence paired with the translation (cross). with the mono prompts we attempt to replicate the evaluation enabled by the gender-neutrality classifier introduced in @xcite on new languages, without generating dedicated data and finetuning new models. mono prompts can be used as is to evaluate intra-lingual neutral rewriting tasks @xcite @xcite . however, for gnt evaluation, they still require gold labels specifying whether the source sentence should be translated neutrally, as with classifier-based methods. the cross prompts instead task the models not just with classifying the target text as gendered or neutral, but also to determine if the target's gender correctly aligns with the source sentence. this allows for gnt evaluation in realistic scenarios, i.e. outside of the benchmarking enabled by gold source sentence labels. for both the mono and cross approaches we experiment with one prompt that requires the model to generate a sentence-level label only (l) and another where the model must generate phrase-level annotations first and then provide a sentence-level label (p+l). examples of the output of each prompt are available in set-g src (f) madam president, i should like to thank mrs oostlander for her sterling contribution as delegate. de ref-g frau präsidentin! ich möchte der kollegin oostlander für ihre verdienstvolle arbeit als delegierte danken. de ref-n geehrtes präsidium! ich möchte dem kollegiumsmitglied oostlander für seine verdienstvolle arbeit als delegierte danken. es ref-g señora presidenta, quiero agradecer a la sra. oostlander sus valiosos esfuerzos como delegada. es ref-n con la venia de la presidencia, quiero agradecer a su señoría oostlander sus valiosos esfuerzos como integrante de la delegación. it ref-g signora presidente, ringrazio la onorevole oostlander per il lavoro meritorio che ha assolto come delegata. it ref-n gentile presidente, ringrazio l'onorevole oostlander per il lavoro meritorio che ha assolto come membro della delegazione. set-n src there are no better guardians of the treaties than the european citizens. de ref-g niemand eignet sich als hüter der verträge besser als die europäischen bürger. de ref-n niemand eignet sich zum hüten der verträge besser als die europäische bevölkerung. es ref-g no hay mejores custodios de los tratados que los ciudadanos europeos. es ref-n no hay mejores vigilantes de los tratados que la ciudadanía europea. it ref-g i migliori guardiani dei trattati sono gli stessi cittadini europei. it ref-n le popolazioni residenti sul suolo europeo sono le migliori custodi dei trattati. ## experimental settings we experiment with llm-based evaluation of gnt from english into three target languages-italian, spanish, and german-in two scenarios: ## results and discussion we report the results of our experiments in targetonly and source-target gnt evaluation experiments on mgente references in figures 1 and 3 respectively. results on automatic gnts are reported in figure 2 . in the target-only charts we include the performance of the gender-neutrality classifier foot_9 on the test data as a baseline for the italian experiments. the detailed results of all evaluation experiments are reported in appendix b, along with additional discussions. to make the performance of the mono and cross prompts comparable in the target-only scenario, we count the labels correctly gendered and wrongly gendered as correct matches of gendered . ## limitations naturally, our work comes with some limitations. first, while the explored evaluation approaches address some of the limitations of previous solutions, they are still confined to discrete sentence-level labels. these make our approaches unable to distinguish degrees of success or failure in using the appropriate gender expression in relation to specific human referents. for instance, this means that we are not able to assess whether a non-neutral output includes only one gendered mention of a human entity or multiple ones, thus we cannot perform more nuanced analyses or rankings of different outputs or systems. second, our approaches do not factor in an important aspect of gnt: the ac-ceptability of neutral text @xcite . acceptability is a complex aspect, determined by the adequacy of the target text with respect to the source sentence meaning and its fluency in the target language. developing evaluation systems that can account for acceptability in the evaluation calls for dedicated research work, and the validation of such systems requires fine-grained human reference annotations that are currently not available. for similar reasons, in our analyses we only focused on the sentence-level annotations generated by the models, and leaving aside the phrase-level annotations generated with the p+l prompts. since the mgente corpus does not include gold annotations of phrases that refer to human beings we only measured sentence-level accuracy and could not evaluate the phrase-level annotations generated by the models. while we are interested in exploring all the aspects mentioned above in the future, with this work we focused on tackling limitations of previously available solutions, enabling and improving the evaluation of gnt across new languages, and doing so with an easy to replicate method, so as to foster the development and research of gnt in further languages as well.