How Reliable is Multilingual LLM -as-a-Judge? (2025)

Abstract: LLM-as-a-Judge has emerged as a popular evaluation strategy, where advanced large language models assess generation results in alignment with human instructions. While these models serve as a promising alternative to human annotators, their reliability in multilingual evaluation remains uncertain. To bridge this gap, we conduct a comprehensive analysis of multilingual LLM-as-a-Judge. Specifically, we evaluate five models from different model families across five diverse tasks involving 25 languages. Our findings reveal that LLMs struggle to achieve consistent judgment results across languages, with an average Fleiss’ Kappa of approximately 0.3, and some models performing even worse. To investigate the cause of inconsistency, we analyze various influencing factors. We observe that consistency varies significantly across languages, with particularly poor performance in low-resource languages. Additionally, we find that neither training on multilingual data nor increasing model scale directly improves judgment consistency. These findings suggest that LLMs are not yet reliable for evaluating multilingual predictions. Our work provides valuable insights into the limitations of multilingual LLM-as-a-Judge, and sheds light on future research.

## introduction the success of various approaches based on neural networks has inspired the development of robust evaluation methods to track advances in the field of nlp @xcite . evaluation aims to assess the quality and performance of nlp models, typically performed using evaluation metrics. prior metrics vary depending on tasks and evaluation aspects, such as accuracy and f1score for classification tasks, and bleu @xcite and rouge @xcite for generation tasks. while these metrics benefit evaluations all question-answer pairs are parallel and perfectly aligned across languages. human evaluators assess the results with uniform criteria. in contrast, llm-as-a-judge demonstrates inconsistency in its judgments, failing to maintain consistency across languages. for various downstream tasks, their reliance on human-annotated references and n-gram matching limits their flexibility and effectiveness. with the development of deep learning, pre-trained language model-based evaluations are introduced, such as bleurt @xcite and bartscore @xcite . they assess output quality by using pre-trained language model representations and generation probability. to offer more efficient and powerful evaluation, some researchers propose llm-as-a-judge @xcite @xcite , which use powerful llms such as gpt4 @xcite to evaluate generated response. @xcite defined evaluation schemes in the prompt template, and rely on existing llms as a judge to offer an evaluation. to avoid the high cost and potential data leakage, @xcite fine-tunes llms as their local evaluators. existing works @xcite show that the result of llm evaluation is consistent with the results ob-tained by expert human evaluation. these methods are subsequently applied to the evaluation of various tasks @xcite . given its superior performance, llm-as-a-judge has been extended to multilingual scenarios, where llms are expected to evaluate responses across different languages @xcite . however, whether llm-as-a-judge is truly trustworthy for multilingual evaluation remains uncertain. a reliable multilingual judge should be consistent, i.e., its judgments should depend on the content of the response rather than the language in which it is presented. figure 1 illustrates a multilingual question answering example, where question-answer pairs are parallel across various languages. a human annotator evaluates these responses consistently, without being influenced by language differences. to assess the reliability of multilingual llm-as-a-judge, we collect five datasets covering different tasks, each with parallel data across multiple languages. we evaluate five models and find that, despite achieving reasonable accuracy within each task, they all struggle to maintain consistent judgments across languages. to further understand the factors affecting consistency, we analyze results across different dimensions. notably, we observe that consistency scores for low-resource languages are significantly lower, even for multilingual llms designed for strong cross-lingual performance, such as aya-expanse @xcite . furthermore, we find that the llm's judgment consistency is influenced by its task-specific ability, highlighting the need to consider the alignment between the evaluation task and the model's domain expertise. overall, our findings shed light on the challenges of using llm-as-a-judge in multilingual settings and provide insights for future research on improving its reliability. our main contributions are as following: you are an ai assistant whose purpose is to evaluate the correctness of answers to questions @xmath0. given a context, a question, and an answer, your goal is to judge whether the generated answer is correct according to the provided context. your evaluation should consider correctness and helpfulness. do not allow the length of the responses to influence evaluation. do not favor certain names of the assistants. be as objective as possible. please format your response as follows: <@xmath1. use one or two sentences at most. keep explaination as concise as possible.]</@xmath2: <@xmath3: <@xmath4: <@xmath5. ## what factors cause inconsistency? to further understand the inferior consistency of multilingual llm-as-a-judge observed in the main results, we investigate potential causes in this section. ## conclusion in this paper, we conduct an in-depth analysis of multilingual llm-as-a-judge, focusing on the consistency of its judgments across parallel data in different languages. our results show that even advanced llms struggle with consistent judgment, exhibiting significant variance across languages. moreover, neither larger model scales nor specific multilingual training improves judgment reliability. our comprehensive analysis provides novel insights into multilingual llm-as-a-judge.